Jul 30 16:58:23.647946 ip-10-0-142-31 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 30 16:58:23.647957 ip-10-0-142-31 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 30 16:58:23.647964 ip-10-0-142-31 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 30 16:58:23.648175 ip-10-0-142-31 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 30 16:58:33.876804 ip-10-0-142-31 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 30 16:58:33.876823 ip-10-0-142-31 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ccbf2a4f86204dbb928047151f3ad035 --
Jul 30 17:00:31.827534 ip-10-0-142-31 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 17:00:32.298543 ip-10-0-142-31 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:32.298543 ip-10-0-142-31 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 17:00:32.298543 ip-10-0-142-31 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:32.298543 ip-10-0-142-31 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 17:00:32.298543 ip-10-0-142-31 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 17:00:32.300855 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.300724    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 17:00:32.304838 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304819    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:32.304838 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304836    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:32.304838 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304840    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304844    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304847    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304850    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304853    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304856    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304861    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304864    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304866    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304869    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304872    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304876    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304879    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304882    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304885    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304888    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304891    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304893    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304897    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304899    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:32.304945 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304902    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304905    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304908    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304911    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304914    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304916    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304920    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304923    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304927    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304932    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304935    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304937    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304940    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304945    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304949    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304953    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304957    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304961    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304964    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304967    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:32.305478 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304970    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304972    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304976    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304980    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304984    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304987    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304990    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304994    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.304997    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305001    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305004    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305007    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305010    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305013    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305015    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305018    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305021    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305023    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305027    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:32.306030 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305030    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305033    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305036    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305038    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305041    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305044    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305047    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305051    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305054    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305057    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305060    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305063    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305066    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305069    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305071    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305074    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305077    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305080    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305083    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305085    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:32.306545 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305088    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305091    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305094    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305097    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305099    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305103    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305106    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305109    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305112    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305115    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305117    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305121    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305124    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305127    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305130    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305133    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305135    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305139    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305940    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305949    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:32.307079 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305952    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305956    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305960    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305964    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305967    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305970    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305973    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305977    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305980    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305983    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305986    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305988    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305991    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305994    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305997    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.305999    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306002    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306005    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306008    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306011    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:32.307620 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306013    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306016    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306019    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306022    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306027    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306031    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306034    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306038    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306041    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306044    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306046    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306049    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306052    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306055    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306057    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306060    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306063    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306066    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306069    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:32.308155 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306072    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306075    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306079    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306082    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306085    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306089    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306093    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306096    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306099    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306101    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306104    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306107    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306110    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306113    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306115    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306118    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306122    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306126    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306129    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:32.308694 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306131    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306134    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306137    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306139    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306142    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306145    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306147    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306150    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306153    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306155    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306158    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306162    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306166    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306170    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306173    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306176    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306180    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306183    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306185    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306188    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:32.309195 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306191    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306194    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306197    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306200    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306202    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306205    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306208    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306211    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306213    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306217    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306219    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306222    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306225    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306228    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306231    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306234    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306236    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306239    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306242    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:32.309730 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306667    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306673    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306676    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306679    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306683    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306686    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306689    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306692    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306695    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306698    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306700    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306703    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306706    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306710    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306712    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306715    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306718    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306721    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306724    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306727    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:32.310250 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306730    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306733    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306736    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306738    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306742    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306745    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306747    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306750    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306753    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306756    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306761    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306765    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306768    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306771    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306774    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306777    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306781    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306785    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306788    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:32.310785 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306791    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306794    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306797    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306800    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306803    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306806    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306809    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306811    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306814    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306816    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306819    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306822    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306824    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306827    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306830    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306833    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306836    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306840    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306843    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306845    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:32.311289 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306848    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306851    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306853    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306856    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306859    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306861    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306864    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306867    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306870    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306872    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306875    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306878    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306881    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306884    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306886    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306889    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306892    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306894    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306897    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:32.311877 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306900    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306903    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306905    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306908    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306911    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306916    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306919    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306925    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306928    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306930    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306933    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306936    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306939    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306941    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306944    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306947    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306950    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306952    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306955    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:32.312390 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306958    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.306961    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307039    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307046    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307058    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307065    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307073    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307079    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307087    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307094    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307098    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307102    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307105    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307109    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307112    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307116    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307119    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307122    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307125    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307128    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307134    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307137    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307141    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307144    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 17:00:32.312890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307148    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307151    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307155    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307158    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307161    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307164    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307167    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307170    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307174    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307177    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307181    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307185    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307188    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307191    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307194    2569 flags.go:64] FLAG: --enable-server="true"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307198    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307202    2569 flags.go:64] FLAG: --event-burst="100"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307205    2569 flags.go:64] FLAG: --event-qps="50"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307209    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307212    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307215    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307219    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307222    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307225    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307228    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 30 17:00:32.313564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307232    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307235    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307238    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307244    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307247    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307250    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307254    2569 flags.go:64] FLAG: --feature-gates=""
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307258    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307261    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307265    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307268    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307271    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307274    2569 flags.go:64] FLAG: --help="false"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307278    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307281    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307284    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307287    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307291    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307295    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307298    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307301    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307304    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307307    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 17:00:32.314217 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307310    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307314    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307317    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307320    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307322    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307326    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307344    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307350    2569 flags.go:64] FLAG: --lock-file=""
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307355    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307359    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307362    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307368    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307374    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307377    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307381    2569 flags.go:64] FLAG: --logging-format="text"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307384    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307388    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307391    2569 flags.go:64] FLAG: --manifest-url=""
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307394    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307399    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307402    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307406    2569 flags.go:64] FLAG: --max-pods="110"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307409    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307412    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307415    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 17:00:32.314869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307418    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307421    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307425    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307428    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307435    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307439    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307442    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307445    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307449    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307455    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307458    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307462    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307465    2569 flags.go:64] FLAG: --port="10250"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307468    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307471    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b8852942ee1c9a70"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307474    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307478    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307481    2569 flags.go:64] FLAG: --register-node="true"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307484    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307489    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307494    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307497    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307500    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307504    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307508    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 17:00:32.315545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307511    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307514    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307517    2569 flags.go:64] FLAG: --runonce="false"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307520    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307524    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307527    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307530    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307533    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307536    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307539    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307543    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307546    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307548    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307551    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307554    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307558    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307562    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307565    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307570    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307574    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307577    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307581    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307583    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307586    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307589    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 17:00:32.316286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307593    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307598    2569 flags.go:64] FLAG: --v="2"
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307603    2569 flags.go:64] FLAG: --version="false"
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307607    2569 flags.go:64] FLAG: --vmodule=""
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307611    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.307615    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307747    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307752    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307755    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307758    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307762    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307765    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307767    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307770    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307773    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307776    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307778    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307781    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307783    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307786    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307789    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:32.317089 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307793    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307797    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307800    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307803    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307806    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307809    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307811    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307814    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307817    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307819    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307822    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307825    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307828    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307832    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307835    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307838    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307841    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307844    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307847    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307850    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:32.317672 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307852    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307855    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307858    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307861    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307863    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307866    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307869    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307872    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307875    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307878    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307881    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307883    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307886    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307889    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307891    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307894    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307898    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307900    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307903    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307906    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:32.318369 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307909    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307913    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307916    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307919    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307922    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307927    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307930    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307933    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307936    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307939    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307942    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307945    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307948    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307951    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307953    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307956    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307959    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307962    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307964    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307967    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:32.319101 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307970    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307972    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307975    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307978    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307981    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307984    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307986    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307989    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307992    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307995    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.307998    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308001    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308004    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308007    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308009    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308012    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308015    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308019    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308021    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308024    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:32.319689 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308027    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308031    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308035    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.308039    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.308045    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.315728    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.315747    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.316756    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.316764    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316860    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316865    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316869    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316872    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316875    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:32.320211 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316878    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316881    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316885    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316888    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316891    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316894    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316896    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316899    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316903    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316905    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316908    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316911    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316913    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316916    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316919    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316922    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316925    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316927    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316931    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316934    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:32.320633 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316937    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316940    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316942    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316945    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316954    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316958    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316963    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316966    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316970    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316973    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316976    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316979    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316982    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316985    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316987    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316990    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316993    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316995    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.316998    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:32.321161 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317001    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317004    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317006    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317009    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317012    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317017    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317020    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317023    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317027    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317031    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317034    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317038    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317041    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317044    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317047    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317050    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317053    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317055    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317064    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:32.321671 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317067    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317070    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317072    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317075    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317077    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317080    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317083    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317086    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317088    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317091    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317094    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317097    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317100    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317103    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317106    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317109    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317111    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317114    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317117    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317119    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:32.322167 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317122    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317125    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317127    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317130    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317133    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317136    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317139    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317141    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317144    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317147    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317149    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317152    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317161    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317164    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317166    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:32.322704 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317169    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.317174    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317373    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317379    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317382    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317385    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317388    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317391    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317394    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317397    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317400    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317403    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317406    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317409    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317413    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 17:00:32.323081 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317416    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317419    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317422    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317425    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317427    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317430    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317433    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317436    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317439    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317442    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317445    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317448    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317451    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317454    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317456    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317465    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317468    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317472    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317476    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317480    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 17:00:32.323516 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317483    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317486    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317489    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317492    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317494    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317497    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317500    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317502    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317505    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317507    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317510    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317513    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317515    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317518    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317521    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317524    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317527    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317529    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317532    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317535    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 17:00:32.324025 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317537    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317540    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317543    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317546    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317548    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317551    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317554    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317556    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317568    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317571    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317574    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317576    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317579    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317582    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317585    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317588    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317590    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317593    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317596    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 17:00:32.324577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317598    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317601    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317604    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317606    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317609    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317612    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317614    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317617    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317620    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317623    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317626    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317630    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317633    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317636    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317639    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317641    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317644    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317647    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317649    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317652    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 17:00:32.325070 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317654    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317657    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317672    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317676    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317680    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317683    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:32.317687    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.317692    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.319721    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.323479    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.325010    2569 server.go:1019] "Starting client certificate rotation"
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.325121    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 17:00:32.325643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.325157    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 17:00:32.327882 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.327868    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 17:00:32.329097 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.329076    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 17:00:32.339627 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.339606    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 30 17:00:32.341873 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.341860    2569 log.go:25] "Validated CRI v1 image API"
Jul 30 17:00:32.343589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.343572    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 17:00:32.348484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.348462    2569 fs.go:135] Filesystem UUIDs: map[5bce1c0f-93b7-47a1-9a4a-164e56e28070:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ff75d162-5caf-432b-a86a-af55f171fbb8:/dev/nvme0n1p4]
Jul 30 17:00:32.348548 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.348482    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 17:00:32.356185 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.356074    2569 manager.go:216] Machine: {Timestamp:2026-07-30 17:00:32.354359588 +0000 UTC m=+0.387169921 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103724 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd31deb51fc1e916501254eaec6a4 SystemUUID:ec2bd31d-eb51-fc1e-9165-01254eaec6a4 BootID:ccbf2a4f-8620-4dbb-9280-47151f3ad035 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:5f:4c:93:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:5f:4c:93:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:59:59:d4:b8:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 17:00:32.356185 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.356180    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 17:00:32.356296 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.356254    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 17:00:32.359435 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.359405    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 17:00:32.359597 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.359439    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 17:00:32.360232 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.360222    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 17:00:32.360264 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.360235    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 17:00:32.360264 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.360244    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 17:00:32.360264 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.360257    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 17:00:32.361269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.361249    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 17:00:32.361944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.361931    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 17:00:32.365421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365409    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 17:00:32.365534 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365525    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 17:00:32.365580 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365572    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 17:00:32.365611 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365590    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 17:00:32.365675 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365664    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 17:00:32.365712 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365684    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 30 17:00:32.365758 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.365716    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 17:00:32.366156 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.366139    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 17:00:32.366156 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.366161    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 17:00:32.369072 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.369051    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 17:00:32.370819 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.370801    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 17:00:32.370893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.370835    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 17:00:32.372015 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.371993    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 17:00:32.372015 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372012    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 17:00:32.372015 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372019    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372025    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372034    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372039    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372048    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372055    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372065    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372072    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372080    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 17:00:32.372197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372086    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 17:00:32.372955 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372944    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 17:00:32.372955 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.372953    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 17:00:32.376213 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.376198    2569 server.go:1268] "Started kubelet"
Jul 30 17:00:32.376387 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.376320    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 17:00:32.376460 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.376410    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 17:00:32.376604 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.376577    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 17:00:32.377036 ip-10-0-142-31 systemd[1]: Started Kubernetes Kubelet.
Jul 30 17:00:32.377686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.377535    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 17:00:32.377686 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.377631    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 17:00:32.377686 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.377633    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 17:00:32.378186 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.378171    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 17:00:32.379341 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.379316    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 17:00:32.383245 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.383215    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 17:00:32.385025 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.383908    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-31.ec2.internal.18c71fe1e6cfe40f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-31.ec2.internal,UID:ip-10-0-142-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-31.ec2.internal,},FirstTimestamp:2026-07-30 17:00:32.376177679 +0000 UTC m=+0.408988009,LastTimestamp:2026-07-30 17:00:32.376177679 +0000 UTC m=+0.408988009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-31.ec2.internal,}"
Jul 30 17:00:32.385773 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.385756    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2zbz"
Jul 30 17:00:32.386007 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.385994    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 17:00:32.386054 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.386039    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 17:00:32.386212 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.386190    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.386758 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.386739    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 17:00:32.386858 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.386773    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 17:00:32.386858 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.386800    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 17:00:32.386973 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.386900    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 17:00:32.386973 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.386910    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 17:00:32.387318 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387287    2569 factory.go:153] Registering CRI-O factory
Jul 30 17:00:32.387405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387330    2569 factory.go:223] Registration of the crio container factory successfully
Jul 30 17:00:32.387405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387383    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 17:00:32.387405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387392    2569 factory.go:55] Registering systemd factory
Jul 30 17:00:32.387405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387401    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 30 17:00:32.387600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387424    2569 factory.go:103] Registering Raw factory
Jul 30 17:00:32.387600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.387441    2569 manager.go:1196] Started watching for new ooms in manager
Jul 30 17:00:32.388412 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.388396    2569 manager.go:349] Starting recovery of all containers
Jul 30 17:00:32.394963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.394936    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x2zbz"
Jul 30 17:00:32.397865 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.397832    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 17:00:32.398016 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.397998    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 17:00:32.401607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.401589    2569 manager.go:354] Recovery completed
Jul 30 17:00:32.405882 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.405870    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.407899 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.407884    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.407957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.407916    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.407957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.407927    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.408467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.408449    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 17:00:32.408467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.408463    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 17:00:32.408467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.408467    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.408651 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.408478    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 17:00:32.409153 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.409137    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.409201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.409166    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.409201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.409179    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.409664 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.409649    2569 policy_none.go:49] "None policy: Start"
Jul 30 17:00:32.409664 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.409666    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 17:00:32.409802 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.409681    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 17:00:32.409861 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.409752    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-31.ec2.internal.18c71fe1e8b3ef8d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-31.ec2.internal,UID:ip-10-0-142-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-31.ec2.internal,},FirstTimestamp:2026-07-30 17:00:32.407900045 +0000 UTC m=+0.440710376,LastTimestamp:2026-07-30 17:00:32.407900045 +0000 UTC m=+0.440710376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-31.ec2.internal,}"
Jul 30 17:00:32.410916 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.410898    2569 policy_none.go:47] "Start"
Jul 30 17:00:32.459071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.458932    2569 manager.go:337] "Starting Device Plugin manager"
Jul 30 17:00:32.459193 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.459095    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 17:00:32.459193 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.459110    2569 server.go:87] "Starting device plugin registration server"
Jul 30 17:00:32.459400 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.459386    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 17:00:32.459461 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.459402    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 17:00:32.459511 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.459484    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 17:00:32.459569 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.459562    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 17:00:32.459622 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.459571    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 17:00:32.460211 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.460171    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 17:00:32.460311 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.460247    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.560243 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.560209    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.561561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.561352    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.561561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.561385    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.561561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.561396    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.561561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.561426    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.568788 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.568770    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.568895 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.568799    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-31.ec2.internal\": node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.585182 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.585157    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.645040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.645009    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 17:00:32.646340 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.646306    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 17:00:32.646447 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.646356    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 17:00:32.646447 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.646376    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 17:00:32.646447 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.646409    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 17:00:32.648634 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.648612    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:32.685238 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.685215    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.746926 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.746841    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"]
Jul 30 17:00:32.747115 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.746961    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.748954 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.748935    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.749057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.748974    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.749057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.748989    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.750876 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.750859    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.751001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.750980    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.751157 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751145    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.751715 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751698    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.751785 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751703    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.751785 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751746    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.751785 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751757    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.751785 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751724    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.751920 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.751798    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.753414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.753400    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.753467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.753429    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 17:00:32.754184 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.754165    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 17:00:32.754251 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.754194    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 17:00:32.754251 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.754204    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeHasSufficientPID"
Jul 30 17:00:32.775979 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.775961    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-31.ec2.internal\" not found" node="ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.780578 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.780562    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-31.ec2.internal\" not found" node="ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.785632 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.785617    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.885795 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.885743    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.888111 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.888094    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef84da010c241ebfac81f7378c666253-config\") pod \"kube-apiserver-proxy-ip-10-0-142-31.ec2.internal\" (UID: \"ef84da010c241ebfac81f7378c666253\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.888175 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.888120    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81aff41b649dded9cc1db78fddd9827a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"81aff41b649dded9cc1db78fddd9827a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.888175 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.888140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aff41b649dded9cc1db78fddd9827a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"81aff41b649dded9cc1db78fddd9827a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.993351 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:32.993308    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:32.993496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.993381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef84da010c241ebfac81f7378c666253-config\") pod \"kube-apiserver-proxy-ip-10-0-142-31.ec2.internal\" (UID: \"ef84da010c241ebfac81f7378c666253\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.993496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.993419    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81aff41b649dded9cc1db78fddd9827a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"81aff41b649dded9cc1db78fddd9827a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.993496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.993449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aff41b649dded9cc1db78fddd9827a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"81aff41b649dded9cc1db78fddd9827a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.993496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.993479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef84da010c241ebfac81f7378c666253-config\") pod \"kube-apiserver-proxy-ip-10-0-142-31.ec2.internal\" (UID: \"ef84da010c241ebfac81f7378c666253\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.993496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.993486    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81aff41b649dded9cc1db78fddd9827a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"81aff41b649dded9cc1db78fddd9827a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:32.993682 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:32.993503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aff41b649dded9cc1db78fddd9827a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal\" (UID: \"81aff41b649dded9cc1db78fddd9827a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:33.083065 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.083014    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:33.084142 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.084103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:33.094014 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:33.093997    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:33.194444 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:33.194367    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:33.294620 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:33.294592    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-31.ec2.internal\" not found"
Jul 30 17:00:33.326134 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.326107    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 17:00:33.326569 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.326265    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 17:00:33.388884 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.388861    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:33.400781 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.400742    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 16:55:33 +0000 UTC" deadline="2028-03-05 21:39:33.487964946 +0000 UTC"
Jul 30 17:00:33.400781 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.400777    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14020h39m0.087191562s"
Jul 30 17:00:33.420711 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.420690    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:33.487884 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.487827    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:33.499526 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.499504    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 17:00:33.500653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.500640    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal"
Jul 30 17:00:33.510074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.510057    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 17:00:33.730024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.729991    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:33.976187 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:33.976147    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef84da010c241ebfac81f7378c666253.slice/crio-37bf37e96e0be599fcc1ae00bdc1d2807a379ca99f06e4271e65af3467d3f146 WatchSource:0}: Error finding container 37bf37e96e0be599fcc1ae00bdc1d2807a379ca99f06e4271e65af3467d3f146: Status 404 returned error can't find the container with id 37bf37e96e0be599fcc1ae00bdc1d2807a379ca99f06e4271e65af3467d3f146
Jul 30 17:00:33.979965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:33.979948    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:00:34.100682 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.100654    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aff41b649dded9cc1db78fddd9827a.slice/crio-da2d82b2098cb0e7ccd051aa97f43b1c5594f8108bb5660ebfec2495aecefc0c WatchSource:0}: Error finding container da2d82b2098cb0e7ccd051aa97f43b1c5594f8108bb5660ebfec2495aecefc0c: Status 404 returned error can't find the container with id da2d82b2098cb0e7ccd051aa97f43b1c5594f8108bb5660ebfec2495aecefc0c
Jul 30 17:00:34.166593 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.166516    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 17:00:34.369707 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.369660    2569 apiserver.go:52] "Watching apiserver"
Jul 30 17:00:34.375222 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.375200    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 17:00:34.375637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.375611    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal","openshift-multus/multus-additional-cni-plugins-swrqn","openshift-multus/multus-hz896","openshift-network-operator/iptables-alerter-vhmwn","kube-system/global-pull-secret-syncer-2w6gg","kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg","openshift-multus/network-metrics-daemon-qfh2d","openshift-network-diagnostics/network-check-target-5gzq2","openshift-ovn-kubernetes/ovnkube-node-l9f99","kube-system/konnectivity-agent-sjw84","openshift-cluster-node-tuning-operator/tuned-c5rf9","openshift-dns/node-resolver-gj2bd","openshift-image-registry/node-ca-llp2c"]
Jul 30 17:00:34.377312 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.377288    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.378607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.378586    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.379693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.379667    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.380135 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.380115    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.380230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.380118    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.380230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.380221    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 17:00:34.380375 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.380247    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kwpcb\""
Jul 30 17:00:34.380885 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.380867    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6pkl2\""
Jul 30 17:00:34.381040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.381017    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.381143 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.381127    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.381602 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.381582    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.382585 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.382563    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.382687 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.382641    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:34.383098 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.383074    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 17:00:34.383220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.383112    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 17:00:34.383290 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.383258    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-58qzt\""
Jul 30 17:00:34.383342 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.383264    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.383403 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.383391    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.384367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.384187    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 17:00:34.384367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.384220    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.384535 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.384504    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2ljqx\""
Jul 30 17:00:34.384694 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.384673    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.384793 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.384759    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.386660 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.386470    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 17:00:34.387191 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.387170    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.387413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.387348    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 17:00:34.387546 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.387507    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.387620 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.387579    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d724j\""
Jul 30 17:00:34.387832 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.387818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:34.387951 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.387898    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:34.389349 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.389252    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:34.389453 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.389348    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:34.390761 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.390741    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.393050 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.393027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.393908 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.393885    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bbpp2\""
Jul 30 17:00:34.394014 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.393951    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 17:00:34.394091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.394069    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 17:00:34.395259 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.394652    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.395488 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.395467    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 17:00:34.395789 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.395772    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rjgv6\""
Jul 30 17:00:34.395873 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.395799    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.396037 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.396020    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 17:00:34.396178 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.396156    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 17:00:34.396413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.396387    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.396551 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.396531    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 17:00:34.397276 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.397240    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 17:00:34.397718 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.397668    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 17:00:34.397947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.397927    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.398018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.397951    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vqrh7\""
Jul 30 17:00:34.401267 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.401244    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 17:00:34.401596 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.401577    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xgpz7\""
Jul 30 17:00:34.402479 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-cni-binary-copy\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.402576 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-daemon-config\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.402576 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402239    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 17:00:34.402576 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402539    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f1a6e1-def7-469b-ad6d-82e62ad3075b-tmp-dir\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.402576 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a918e3e-b901-4f22-b193-4d40e2964878-host\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.402779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6a918e3e-b901-4f22-b193-4d40e2964878-serviceca\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.402779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402648    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-cni-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.402881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5962bc8b-6237-4100-830f-5a54e63e0f14-kubelet-config\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.402881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402838    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmw2\" (UniqueName: \"kubernetes.io/projected/be5a7a49-6ba1-410f-91bc-fa13983a7b94-kube-api-access-7bmw2\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.402976 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-kubelet\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.402976 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-hostroot\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.403129 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.402997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:34.403184 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48054529-4ab8-4319-a00d-bfd1577bddde-host-slash\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.403184 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403159    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-device-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.403278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403183    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.403278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403232    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-cnibin\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.403400 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-cni-multus\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.403400 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403341    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5w7d\" (UniqueName: \"kubernetes.io/projected/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-kube-api-access-v5w7d\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.403519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xrj\" (UniqueName: \"kubernetes.io/projected/33f1a6e1-def7-469b-ad6d-82e62ad3075b-kube-api-access-n5xrj\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.403616 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403600    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 17:00:34.403667 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvs74\" (UniqueName: \"kubernetes.io/projected/48054529-4ab8-4319-a00d-bfd1577bddde-kube-api-access-xvs74\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.403761 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-cni-bin\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.403761 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403705    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-etc-kubernetes\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.403761 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403733    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwww6\" (UniqueName: \"kubernetes.io/projected/6a918e3e-b901-4f22-b193-4d40e2964878-kube-api-access-kwww6\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.403920 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403849    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-socket-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.403977 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-sys-fs\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.404042 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.403983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-socket-dir-parent\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.404042 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33f1a6e1-def7-469b-ad6d-82e62ad3075b-hosts-file\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.404269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404043    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-registration-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.404269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.404269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48054529-4ab8-4319-a00d-bfd1577bddde-iptables-alerter-script\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.404269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404263    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-k8s-cni-cncf-io\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.404495 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404351    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-netns\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.404495 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-conf-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.404495 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404467    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-multus-certs\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.404653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5962bc8b-6237-4100-830f-5a54e63e0f14-dbus\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.404653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.404653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-system-cni-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.404653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.404633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-os-release\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.422836 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.422814    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5srxq"
Jul 30 17:00:34.431684 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.431660    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5srxq"
Jul 30 17:00:34.488321 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.488292    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 17:00:34.505822 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-cni-multus\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.505961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-cnibin\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.505961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505865    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-slash\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.505961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505890    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-systemd\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.505961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-lib-modules\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.505961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-cni-multus\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.505943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-cni-bin\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwww6\" (UniqueName: \"kubernetes.io/projected/6a918e3e-b901-4f22-b193-4d40e2964878-kube-api-access-kwww6\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-cni-bin\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-socket-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-socket-dir-parent\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33f1a6e1-def7-469b-ad6d-82e62ad3075b-hosts-file\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-system-cni-dir\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovnkube-script-lib\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506142    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-registration-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506179    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-cni-binary-copy\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-ovn\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.506230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-k8s-cni-cncf-io\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-netns\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506369    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-k8s-cni-cncf-io\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-netns\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33f1a6e1-def7-469b-ad6d-82e62ad3075b-hosts-file\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506473    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-registration-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5962bc8b-6237-4100-830f-5a54e63e0f14-dbus\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-socket-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-var-lib-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.506574    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506593    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-etc-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506611    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-host\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.506665    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:35.006640195 +0000 UTC m=+3.039450527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-os-release\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506719    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-daemon-config\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506761    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f1a6e1-def7-469b-ad6d-82e62ad3075b-tmp-dir\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.506947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-os-release\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-systemd\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506689    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-node-log\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.506999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-cni-netd\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6a918e3e-b901-4f22-b193-4d40e2964878-serviceca\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-cni-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zjl\" (UniqueName: \"kubernetes.io/projected/ffa93763-d018-4c80-aa34-83bc592bc8b8-kube-api-access-n7zjl\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507087    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507121    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5962bc8b-6237-4100-830f-5a54e63e0f14-dbus\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysctl-conf\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507153    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-run\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-cni-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmw2\" (UniqueName: \"kubernetes.io/projected/be5a7a49-6ba1-410f-91bc-fa13983a7b94-kube-api-access-7bmw2\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48054529-4ab8-4319-a00d-bfd1577bddde-host-slash\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507204    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-os-release\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.507925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507254    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovn-node-metrics-cert\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-sys\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-var-lib-kubelet\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507317    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7e5b982-a2b4-439a-8352-3966c1439ce0-tmp\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507363    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-device-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5w7d\" (UniqueName: \"kubernetes.io/projected/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-kube-api-access-v5w7d\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48054529-4ab8-4319-a00d-bfd1577bddde-host-slash\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f1a6e1-def7-469b-ad6d-82e62ad3075b-tmp-dir\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507456    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xrj\" (UniqueName: \"kubernetes.io/projected/33f1a6e1-def7-469b-ad6d-82e62ad3075b-kube-api-access-n5xrj\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvs74\" (UniqueName: \"kubernetes.io/projected/48054529-4ab8-4319-a00d-bfd1577bddde-kube-api-access-xvs74\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507696    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507774    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-modprobe-d\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-tuned\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507843    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-etc-kubernetes\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507874    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-kubernetes\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507887    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-socket-dir-parent\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.508687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-sys-fs\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-cni-bin\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.507984    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysconfig\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tq4\" (UniqueName: \"kubernetes.io/projected/b7e5b982-a2b4-439a-8352-3966c1439ce0-kube-api-access-86tq4\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48054529-4ab8-4319-a00d-bfd1577bddde-iptables-alerter-script\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508108    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-etc-kubernetes\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508110    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6a918e3e-b901-4f22-b193-4d40e2964878-serviceca\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508157    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508178    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-device-dir\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508194    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8l5\" (UniqueName: \"kubernetes.io/projected/072815d7-8920-4d28-9e01-95e3a9908d71-kube-api-access-gc8l5\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-kubelet\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508326    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysctl-d\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-conf-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508397    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-multus-certs\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-log-socket\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.509497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508535    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d27bd079-abf6-40c3-a09b-eca4993d05ed-agent-certs\") pod \"konnectivity-agent-sjw84\" (UID: \"d27bd079-abf6-40c3-a09b-eca4993d05ed\") " pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-system-cni-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508594    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-cni-binary-copy\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508623    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-systemd-units\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a918e3e-b901-4f22-b193-4d40e2964878-host\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508697    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/48054529-4ab8-4319-a00d-bfd1577bddde-iptables-alerter-script\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508710    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5962bc8b-6237-4100-830f-5a54e63e0f14-kubelet-config\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-run-netns\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508779    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5962bc8b-6237-4100-830f-5a54e63e0f14-kubelet-config\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d27bd079-abf6-40c3-a09b-eca4993d05ed-konnectivity-ca\") pod \"konnectivity-agent-sjw84\" (UID: \"d27bd079-abf6-40c3-a09b-eca4993d05ed\") " pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-kubelet\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-sys-fs\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-hostroot\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovnkube-config\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508974    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-daemon-config\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-system-cni-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.510357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.508972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-env-overrides\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509060    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-multus-conf-dir\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-var-lib-kubelet\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509130    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-hostroot\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509136    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-host-run-multus-certs\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a918e3e-b901-4f22-b193-4d40e2964878-host\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cq9\" (UniqueName: \"kubernetes.io/projected/42f582b9-f05b-4c3e-996f-eebb4118a05a-kube-api-access-k2cq9\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-cnibin\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509461    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-cnibin\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/be5a7a49-6ba1-410f-91bc-fa13983a7b94-etc-selinux\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.511091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.509538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-cni-binary-copy\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.517902 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.517874    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 17:00:34.517991 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.517927    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:34.517991 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.517947    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:34.517991 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.517961    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:34.518123 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.518051    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:35.018037747 +0000 UTC m=+3.050848063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:34.522234 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.522207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwww6\" (UniqueName: \"kubernetes.io/projected/6a918e3e-b901-4f22-b193-4d40e2964878-kube-api-access-kwww6\") pod \"node-ca-llp2c\" (UID: \"6a918e3e-b901-4f22-b193-4d40e2964878\") " pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.522352 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.522218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5w7d\" (UniqueName: \"kubernetes.io/projected/3d02675c-5b7b-4e38-a9a3-6aff3bd89563-kube-api-access-v5w7d\") pod \"multus-hz896\" (UID: \"3d02675c-5b7b-4e38-a9a3-6aff3bd89563\") " pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.523013 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.522987    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmw2\" (UniqueName: \"kubernetes.io/projected/be5a7a49-6ba1-410f-91bc-fa13983a7b94-kube-api-access-7bmw2\") pod \"aws-ebs-csi-driver-node-cswfg\" (UID: \"be5a7a49-6ba1-410f-91bc-fa13983a7b94\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.523766 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.523716    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xrj\" (UniqueName: \"kubernetes.io/projected/33f1a6e1-def7-469b-ad6d-82e62ad3075b-kube-api-access-n5xrj\") pod \"node-resolver-gj2bd\" (UID: \"33f1a6e1-def7-469b-ad6d-82e62ad3075b\") " pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.524425 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.524402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvs74\" (UniqueName: \"kubernetes.io/projected/48054529-4ab8-4319-a00d-bfd1577bddde-kube-api-access-xvs74\") pod \"iptables-alerter-vhmwn\" (UID: \"48054529-4ab8-4319-a00d-bfd1577bddde\") " pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.610382 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-os-release\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-systemd\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-node-log\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-cni-netd\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zjl\" (UniqueName: \"kubernetes.io/projected/ffa93763-d018-4c80-aa34-83bc592bc8b8-kube-api-access-n7zjl\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610541    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysctl-conf\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.610579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-run\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovn-node-metrics-cert\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610642    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-sys\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-var-lib-kubelet\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610679    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-cni-netd\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7e5b982-a2b4-439a-8352-3966c1439ce0-tmp\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610751    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610820    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-modprobe-d\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-tuned\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-kubernetes\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-cni-bin\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.610984    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysconfig\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tq4\" (UniqueName: \"kubernetes.io/projected/b7e5b982-a2b4-439a-8352-3966c1439ce0-kube-api-access-86tq4\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611065    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:34.611089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8l5\" (UniqueName: \"kubernetes.io/projected/072815d7-8920-4d28-9e01-95e3a9908d71-kube-api-access-gc8l5\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-kubelet\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611195    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611223    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysctl-d\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-log-socket\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d27bd079-abf6-40c3-a09b-eca4993d05ed-agent-certs\") pod \"konnectivity-agent-sjw84\" (UID: \"d27bd079-abf6-40c3-a09b-eca4993d05ed\") " pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-systemd-units\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-run-netns\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611430    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d27bd079-abf6-40c3-a09b-eca4993d05ed-konnectivity-ca\") pod \"konnectivity-agent-sjw84\" (UID: \"d27bd079-abf6-40c3-a09b-eca4993d05ed\") " pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611494    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovnkube-config\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.611913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611524    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613311 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.611590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-kubernetes\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.613445 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-cni-bin\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613445 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613404    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-sys\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.613445 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-run\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.613598 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-env-overrides\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613598 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cq9\" (UniqueName: \"kubernetes.io/projected/42f582b9-f05b-4c3e-996f-eebb4118a05a-kube-api-access-k2cq9\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613598 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-cnibin\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.613755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613605    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-slash\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613639    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-systemd\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.613755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613675    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-lib-modules\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.613755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-system-cni-dir\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.613960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovnkube-script-lib\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-cni-binary-copy\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.613960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613829    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-ovn\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613850    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-var-lib-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613871    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-etc-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.613960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-host\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.614273 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.613992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-host\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.614358 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-var-lib-kubelet\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.614421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614398    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-slash\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.614421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-systemd\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.614421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-cnibin\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.614606 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-lib-modules\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.614674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614657    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-system-cni-dir\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.614777 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614755    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-tuned\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.614960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.614942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615145    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysconfig\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615216    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/072815d7-8920-4d28-9e01-95e3a9908d71-os-release\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615365    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-systemd\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysctl-conf\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615404    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovnkube-script-lib\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615452    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-node-log\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-modprobe-d\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.615632 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-run-ovn\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616096 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-var-lib-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616096 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.615997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.616096 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.616058    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:34.616096 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-etc-openvswitch\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616308 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:34.616140    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:35.116118369 +0000 UTC m=+3.148928707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:34.616308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616136    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovn-node-metrics-cert\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-systemd-units\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-log-socket\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616586 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616545    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-run-netns\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.616697 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.617410 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.617387    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d27bd079-abf6-40c3-a09b-eca4993d05ed-konnectivity-ca\") pod \"konnectivity-agent-sjw84\" (UID: \"d27bd079-abf6-40c3-a09b-eca4993d05ed\") " pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.617513 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.617458    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-ovnkube-config\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.618878 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.617625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7e5b982-a2b4-439a-8352-3966c1439ce0-etc-sysctl-d\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.618878 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.617930    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/42f582b9-f05b-4c3e-996f-eebb4118a05a-env-overrides\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.619138 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.619052    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.619138 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.616747    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/42f582b9-f05b-4c3e-996f-eebb4118a05a-host-kubelet\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.619550 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.619465    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/072815d7-8920-4d28-9e01-95e3a9908d71-cni-binary-copy\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.622430 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.622407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7e5b982-a2b4-439a-8352-3966c1439ce0-tmp\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.623830 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.623793    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cq9\" (UniqueName: \"kubernetes.io/projected/42f582b9-f05b-4c3e-996f-eebb4118a05a-kube-api-access-k2cq9\") pod \"ovnkube-node-l9f99\" (UID: \"42f582b9-f05b-4c3e-996f-eebb4118a05a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.626569 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.626156    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d27bd079-abf6-40c3-a09b-eca4993d05ed-agent-certs\") pod \"konnectivity-agent-sjw84\" (UID: \"d27bd079-abf6-40c3-a09b-eca4993d05ed\") " pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.629076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.629049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zjl\" (UniqueName: \"kubernetes.io/projected/ffa93763-d018-4c80-aa34-83bc592bc8b8-kube-api-access-n7zjl\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:34.629693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.629675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tq4\" (UniqueName: \"kubernetes.io/projected/b7e5b982-a2b4-439a-8352-3966c1439ce0-kube-api-access-86tq4\") pod \"tuned-c5rf9\" (UID: \"b7e5b982-a2b4-439a-8352-3966c1439ce0\") " pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.633864 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.633811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8l5\" (UniqueName: \"kubernetes.io/projected/072815d7-8920-4d28-9e01-95e3a9908d71-kube-api-access-gc8l5\") pod \"multus-additional-cni-plugins-swrqn\" (UID: \"072815d7-8920-4d28-9e01-95e3a9908d71\") " pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.652861 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.652799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" event={"ID":"81aff41b649dded9cc1db78fddd9827a","Type":"ContainerStarted","Data":"da2d82b2098cb0e7ccd051aa97f43b1c5594f8108bb5660ebfec2495aecefc0c"}
Jul 30 17:00:34.655385 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.655311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" event={"ID":"ef84da010c241ebfac81f7378c666253","Type":"ContainerStarted","Data":"37bf37e96e0be599fcc1ae00bdc1d2807a379ca99f06e4271e65af3467d3f146"}
Jul 30 17:00:34.689865 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.689768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg"
Jul 30 17:00:34.698465 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.698282    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gj2bd"
Jul 30 17:00:34.701875 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.701842    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5a7a49_6ba1_410f_91bc_fa13983a7b94.slice/crio-e78a09fc143c4f05c67b6b0e01af3907f7dbf097d258d223446f98c2b8fb2a8c WatchSource:0}: Error finding container e78a09fc143c4f05c67b6b0e01af3907f7dbf097d258d223446f98c2b8fb2a8c: Status 404 returned error can't find the container with id e78a09fc143c4f05c67b6b0e01af3907f7dbf097d258d223446f98c2b8fb2a8c
Jul 30 17:00:34.709073 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.709047    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hz896"
Jul 30 17:00:34.709172 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.709079    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f1a6e1_def7_469b_ad6d_82e62ad3075b.slice/crio-f228ab49621bbe8b91a614786df0b6c4b6ea6d8a1c15d13d5113327c524cab86 WatchSource:0}: Error finding container f228ab49621bbe8b91a614786df0b6c4b6ea6d8a1c15d13d5113327c524cab86: Status 404 returned error can't find the container with id f228ab49621bbe8b91a614786df0b6c4b6ea6d8a1c15d13d5113327c524cab86
Jul 30 17:00:34.718380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.718126    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhmwn"
Jul 30 17:00:34.725552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.725375    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llp2c"
Jul 30 17:00:34.727430 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.727380    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48054529_4ab8_4319_a00d_bfd1577bddde.slice/crio-bc59afe4a41dffae3fd8647ace9dbcade71bfbf18ca06cba92934d1e1852617b WatchSource:0}: Error finding container bc59afe4a41dffae3fd8647ace9dbcade71bfbf18ca06cba92934d1e1852617b: Status 404 returned error can't find the container with id bc59afe4a41dffae3fd8647ace9dbcade71bfbf18ca06cba92934d1e1852617b
Jul 30 17:00:34.733394 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.733372    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-swrqn"
Jul 30 17:00:34.734493 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.734468    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a918e3e_b901_4f22_b193_4d40e2964878.slice/crio-0296b2ae52b00c98053dc2abf1528299fccddb8082a3adea80db581bfcb0b3cd WatchSource:0}: Error finding container 0296b2ae52b00c98053dc2abf1528299fccddb8082a3adea80db581bfcb0b3cd: Status 404 returned error can't find the container with id 0296b2ae52b00c98053dc2abf1528299fccddb8082a3adea80db581bfcb0b3cd
Jul 30 17:00:34.740484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.740447    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:00:34.744132 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.744109    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072815d7_8920_4d28_9e01_95e3a9908d71.slice/crio-03ad400300f284e5b050cd0964f34d951c7768de532d7bbb3b7aa39c2539ae9f WatchSource:0}: Error finding container 03ad400300f284e5b050cd0964f34d951c7768de532d7bbb3b7aa39c2539ae9f: Status 404 returned error can't find the container with id 03ad400300f284e5b050cd0964f34d951c7768de532d7bbb3b7aa39c2539ae9f
Jul 30 17:00:34.747882 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.747691    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:34.749915 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.749889    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f582b9_f05b_4c3e_996f_eebb4118a05a.slice/crio-9fc8213da49ebb0c497d2c73edeb021598dcd9b027ad13aa2f3326f16ce4de20 WatchSource:0}: Error finding container 9fc8213da49ebb0c497d2c73edeb021598dcd9b027ad13aa2f3326f16ce4de20: Status 404 returned error can't find the container with id 9fc8213da49ebb0c497d2c73edeb021598dcd9b027ad13aa2f3326f16ce4de20
Jul 30 17:00:34.754541 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:34.754344    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c5rf9"
Jul 30 17:00:34.757898 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.757872    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27bd079_abf6_40c3_a09b_eca4993d05ed.slice/crio-a727d69701b27dd875df38b1c79c8e4a656c5fd6569f25fcf7f9a89a76990d42 WatchSource:0}: Error finding container a727d69701b27dd875df38b1c79c8e4a656c5fd6569f25fcf7f9a89a76990d42: Status 404 returned error can't find the container with id a727d69701b27dd875df38b1c79c8e4a656c5fd6569f25fcf7f9a89a76990d42
Jul 30 17:00:34.764166 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:00:34.764141    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e5b982_a2b4_439a_8352_3966c1439ce0.slice/crio-717fc9ff78b7d19ccd52bf07482b47a696bf14f3fda6479c2d9d8a9503682ae7 WatchSource:0}: Error finding container 717fc9ff78b7d19ccd52bf07482b47a696bf14f3fda6479c2d9d8a9503682ae7: Status 404 returned error can't find the container with id 717fc9ff78b7d19ccd52bf07482b47a696bf14f3fda6479c2d9d8a9503682ae7
Jul 30 17:00:35.017929 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.017831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:35.018098 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.017990    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:35.018098 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.018065    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:36.018043808 +0000 UTC m=+4.050854125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:35.118465 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.118425    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:35.118639 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.118476    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:35.118639 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.118591    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:35.118639 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.118613    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:35.118639 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.118639    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:35.118819 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.118654    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:35.118819 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.118665    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:36.118644759 +0000 UTC m=+4.151455081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:35.118819 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.118706    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:36.118692812 +0000 UTC m=+4.151503129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:35.434300 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.434202    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:34 +0000 UTC" deadline="2028-02-11 20:46:53.090640145 +0000 UTC"
Jul 30 17:00:35.434300 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.434252    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13467h46m17.656393411s"
Jul 30 17:00:35.647309 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.647055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:35.647309 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:35.647178    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:35.659608 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.659532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sjw84" event={"ID":"d27bd079-abf6-40c3-a09b-eca4993d05ed","Type":"ContainerStarted","Data":"a727d69701b27dd875df38b1c79c8e4a656c5fd6569f25fcf7f9a89a76990d42"}
Jul 30 17:00:35.662390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.662315    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llp2c" event={"ID":"6a918e3e-b901-4f22-b193-4d40e2964878","Type":"ContainerStarted","Data":"0296b2ae52b00c98053dc2abf1528299fccddb8082a3adea80db581bfcb0b3cd"}
Jul 30 17:00:35.664781 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.664717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hz896" event={"ID":"3d02675c-5b7b-4e38-a9a3-6aff3bd89563","Type":"ContainerStarted","Data":"2bfc0de192dba6a62e104c6d504c81647802afe31f8d7097126e3fcf08ebae69"}
Jul 30 17:00:35.668218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.668147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gj2bd" event={"ID":"33f1a6e1-def7-469b-ad6d-82e62ad3075b","Type":"ContainerStarted","Data":"f228ab49621bbe8b91a614786df0b6c4b6ea6d8a1c15d13d5113327c524cab86"}
Jul 30 17:00:35.670687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.670614    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c5rf9" event={"ID":"b7e5b982-a2b4-439a-8352-3966c1439ce0","Type":"ContainerStarted","Data":"717fc9ff78b7d19ccd52bf07482b47a696bf14f3fda6479c2d9d8a9503682ae7"}
Jul 30 17:00:35.674968 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.674868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"9fc8213da49ebb0c497d2c73edeb021598dcd9b027ad13aa2f3326f16ce4de20"}
Jul 30 17:00:35.677657 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.677630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerStarted","Data":"03ad400300f284e5b050cd0964f34d951c7768de532d7bbb3b7aa39c2539ae9f"}
Jul 30 17:00:35.680609 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.680541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhmwn" event={"ID":"48054529-4ab8-4319-a00d-bfd1577bddde","Type":"ContainerStarted","Data":"bc59afe4a41dffae3fd8647ace9dbcade71bfbf18ca06cba92934d1e1852617b"}
Jul 30 17:00:35.687951 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:35.687857    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg" event={"ID":"be5a7a49-6ba1-410f-91bc-fa13983a7b94","Type":"ContainerStarted","Data":"e78a09fc143c4f05c67b6b0e01af3907f7dbf097d258d223446f98c2b8fb2a8c"}
Jul 30 17:00:36.027664 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.026898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:36.027664 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.027156    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:36.027664 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.027232    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:38.02720585 +0000 UTC m=+6.060016177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.128124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.128200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.128377    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.128446    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:38.128425314 +0000 UTC m=+6.161235634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.128550    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.128565    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.128579    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:36.128956 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.128622    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:38.12860822 +0000 UTC m=+6.161418544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:36.435563 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.434822    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 16:55:34 +0000 UTC" deadline="2028-03-30 02:15:52.306273931 +0000 UTC"
Jul 30 17:00:36.435563 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.434867    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14601h15m15.871413296s"
Jul 30 17:00:36.648262 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.647692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:36.648262 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.647830    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:36.648631 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:36.648429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:36.648631 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:36.648565    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:37.647349 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:37.646948    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:37.647349 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:37.647088    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:38.047846 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:38.047802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:38.048067 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.048018    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:38.048160 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.048085    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:42.048064937 +0000 UTC m=+10.080875259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:38.148861 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:38.148821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:38.149068 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:38.148892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:38.149068 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.149028    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:38.149193 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.149097    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:42.149076031 +0000 UTC m=+10.181886350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:38.149270 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.149194    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:38.149270 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.149209    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:38.149270 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.149221    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:38.149270 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.149261    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:42.149246903 +0000 UTC m=+10.182057227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:38.647413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:38.647044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:38.647413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:38.647055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:38.647413 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.647187    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:38.647413 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:38.647327    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:39.647428 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:39.647394    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:39.647937 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:39.647542    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:40.647516 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:40.647423    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:40.647964 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:40.647590    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:40.647964 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:40.647432    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:40.648080 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:40.647984    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:41.646753 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:41.646721    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:41.646942 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:41.646856    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:42.083491 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:42.083449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:42.083945 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.083648    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:42.083945 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.083725    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:50.083704777 +0000 UTC m=+18.116515102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:42.184440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:42.184393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:42.184606 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:42.184500    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:42.184606 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.184555    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:42.184739 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.184636    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:50.184616667 +0000 UTC m=+18.217426995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:42.184739 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.184644    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:42.184739 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.184662    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:42.184739 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.184678    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:42.184739 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.184729    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:00:50.184713682 +0000 UTC m=+18.217524003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:42.648014 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:42.647937    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:42.648182 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.648048    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:42.648459 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:42.648301    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:42.648459 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:42.648414    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:43.647627 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:43.647593    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:43.648110 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:43.647725    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:44.647867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:44.647330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:44.647867 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:44.647464    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:44.647867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:44.647542    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:44.647867 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:44.647660    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:45.646824 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:45.646794    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:45.646977 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:45.646907    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:46.647267 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:46.647225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:46.647705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:46.647231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:46.647705 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:46.647432    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:46.647705 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:46.647571    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:47.647172 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:47.647137    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:47.647365 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:47.647247    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:48.647411 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:48.647379    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:48.647812 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:48.647520    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:48.647812 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:48.647577    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:48.647812 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:48.647709    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:49.647095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:49.647057    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:49.647337 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:49.647183    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:50.140082 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:50.140040    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:50.140566 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.140194    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:50.140566 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.140283    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:06.140250538 +0000 UTC m=+34.173060876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:00:50.240747 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:50.240705    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:50.240931 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:50.240771    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:50.240931 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.240886    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:00:50.240931 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.240912    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:00:50.240931 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.240924    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:50.241192 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.240931    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:50.241192 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.240989    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:06.240971246 +0000 UTC m=+34.273781565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:00:50.241192 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.241010    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:06.241000232 +0000 UTC m=+34.273810553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:00:50.647696 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:50.647481    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:50.647882 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:50.647481    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:50.647882 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.647806    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:50.648004 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:50.647875    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:51.647343 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:51.647305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:51.647758 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:51.647439    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:52.647528 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:52.647482    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:52.647993 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:52.647604    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:52.647993 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:52.647648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:52.647993 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:52.647775    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:53.647506 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:53.647468    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:53.647686 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:53.647600    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:54.647745 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.647363    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:54.647745 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:54.647661    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:54.648321 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.648109    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:54.648321 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:54.648216    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:54.733778 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.733740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" event={"ID":"ef84da010c241ebfac81f7378c666253","Type":"ContainerStarted","Data":"347cbfa7ed19fda04ad88b8023c184eff1cd2e4761ed5a5037a93ea8727f6970"}
Jul 30 17:00:54.735510 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.735480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c5rf9" event={"ID":"b7e5b982-a2b4-439a-8352-3966c1439ce0","Type":"ContainerStarted","Data":"09c324a563c2349b123f5fadb6a05f36615c978286ea2ed3a1adbb37c6328dbf"}
Jul 30 17:00:54.737283 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.737241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"11872ffda8a3ef931ee956a735e14352d108d4318b8e55bd72865b01b6a0c233"}
Jul 30 17:00:54.737393 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.737319    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"8bd4e061ca7e45a7d51664f0c8781181e8dc2373b0021b6057649228d17c0f6d"}
Jul 30 17:00:54.738602 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.738581    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hz896" event={"ID":"3d02675c-5b7b-4e38-a9a3-6aff3bd89563","Type":"ContainerStarted","Data":"270f639d5f8d2c953741d5bf09f1696a681ded12e8a911dce46a1971c238e87f"}
Jul 30 17:00:54.750329 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.749145    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-31.ec2.internal" podStartSLOduration=20.749125863 podStartE2EDuration="20.749125863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:54.74794286 +0000 UTC m=+22.780753200" watchObservedRunningTime="2026-07-30 17:00:54.749125863 +0000 UTC m=+22.781936204"
Jul 30 17:00:54.763329 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:54.763292    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c5rf9" podStartSLOduration=2.327312655 podStartE2EDuration="21.76328086s" totalImagesPullingTime="19.435968205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:54.762964652 +0000 UTC m=+22.795774988" watchObservedRunningTime="2026-07-30 17:00:54.76328086 +0000 UTC m=+22.796091193"
Jul 30 17:00:55.647413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.647383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:55.647586 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:55.647505    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:55.743612 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.743578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"a7c37ee0c4843fe053123f51230d98e10e293a498fcbb2c949c23ce62dc9a81c"}
Jul 30 17:00:55.744372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.743658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"fb9b81b812ff3055b1bea5f0a394da4720a449e95c125ab6fbe73ff982d349b3"}
Jul 30 17:00:55.744372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.743688    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"74c26e18032ac1fffe9e186e6bbe7e2f3c5e44b43857c6a6aac57ef4d50c1069"}
Jul 30 17:00:55.744372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.743726    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"1442725b1bcf17848635ca121e6ccefd41212589f7421cda58c3b93d22efd4a4"}
Jul 30 17:00:55.744981 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.744955    2569 generic.go:356] "Generic (PLEG): container finished" podID="072815d7-8920-4d28-9e01-95e3a9908d71" containerID="ea8eec6a2bd48eabc3d29906df666697e05fd1306940aa28f92c3dc78a690658" exitCode=0
Jul 30 17:00:55.745067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.745042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerDied","Data":"ea8eec6a2bd48eabc3d29906df666697e05fd1306940aa28f92c3dc78a690658"}
Jul 30 17:00:55.746385 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.746361    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhmwn" event={"ID":"48054529-4ab8-4319-a00d-bfd1577bddde","Type":"ContainerStarted","Data":"acabdcc22d87886f3e5af31321905ea1b0a4c138efaf7a5c1842a8ff9bf1d608"}
Jul 30 17:00:55.747950 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.747929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg" event={"ID":"be5a7a49-6ba1-410f-91bc-fa13983a7b94","Type":"ContainerStarted","Data":"0cb29d8bdf62ee56519464097b13929855809256ba95e3c3922663d08029accc"}
Jul 30 17:00:55.749279 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.749242    2569 generic.go:356] "Generic (PLEG): container finished" podID="81aff41b649dded9cc1db78fddd9827a" containerID="6a0b3499a818c5a364d145476486edeef44d9f2ca1d825c9d7dff9f01941a62f" exitCode=0
Jul 30 17:00:55.749385 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.749283    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" event={"ID":"81aff41b649dded9cc1db78fddd9827a","Type":"ContainerDied","Data":"6a0b3499a818c5a364d145476486edeef44d9f2ca1d825c9d7dff9f01941a62f"}
Jul 30 17:00:55.750806 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.750785    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sjw84" event={"ID":"d27bd079-abf6-40c3-a09b-eca4993d05ed","Type":"ContainerStarted","Data":"055b990e5d83216ac8486e4949f0e9e01595801ec5ee13821dc04f9d8514bb19"}
Jul 30 17:00:55.752380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.752345    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llp2c" event={"ID":"6a918e3e-b901-4f22-b193-4d40e2964878","Type":"ContainerStarted","Data":"a13039d560b4a0d7858c3313ca85a815fb34ac288495499f8594e0479a7bc831"}
Jul 30 17:00:55.753579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.753560    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gj2bd" event={"ID":"33f1a6e1-def7-469b-ad6d-82e62ad3075b","Type":"ContainerStarted","Data":"e259dd194adb65d65869dcdd0e13656c2b7bf2d2e91dec0c344a98ee7db0460f"}
Jul 30 17:00:55.769162 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.769131    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hz896" podStartSLOduration=3.26917525 podStartE2EDuration="22.769116665s" totalImagesPullingTime="19.499941415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:54.787293819 +0000 UTC m=+22.820104149" watchObservedRunningTime="2026-07-30 17:00:55.769116665 +0000 UTC m=+23.801927002"
Jul 30 17:00:55.799850 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.799609    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sjw84" podStartSLOduration=7.78023287 podStartE2EDuration="22.799593939s" totalImagesPullingTime="15.019361069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:55.799564001 +0000 UTC m=+23.832374336" watchObservedRunningTime="2026-07-30 17:00:55.799593939 +0000 UTC m=+23.832404277"
Jul 30 17:00:55.800014 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.799976    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-llp2c" podStartSLOduration=3.332573638 podStartE2EDuration="22.79996426s" totalImagesPullingTime="19.467390622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:55.783973424 +0000 UTC m=+23.816783762" watchObservedRunningTime="2026-07-30 17:00:55.79996426 +0000 UTC m=+23.832774597"
Jul 30 17:00:55.814054 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.814022    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gj2bd" podStartSLOduration=3.722631573 podStartE2EDuration="22.814010341s" totalImagesPullingTime="19.091378768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:55.813853375 +0000 UTC m=+23.846663716" watchObservedRunningTime="2026-07-30 17:00:55.814010341 +0000 UTC m=+23.846820679"
Jul 30 17:00:55.827618 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:55.827586    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vhmwn" podStartSLOduration=3.3520899379999998 podStartE2EDuration="22.827573779s" totalImagesPullingTime="19.475483841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:55.827287165 +0000 UTC m=+23.860097501" watchObservedRunningTime="2026-07-30 17:00:55.827573779 +0000 UTC m=+23.860384115"
Jul 30 17:00:56.137373 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.137343    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 17:00:56.183563 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.183536    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:56.184031 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.184009    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:00:56.479826 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.479737    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T17:00:56.137362786Z","UUID":"aa8c9364-4411-4e9f-8656-4eaf5f86c296","Handler":null,"Name":"","Endpoint":""}
Jul 30 17:00:56.481164 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.481143    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 17:00:56.481287 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.481176    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 17:00:56.647433 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.647402    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:56.647603 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.647402    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:56.647603 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:56.647557    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:56.647603 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:56.647595    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:56.759224 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.759112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg" event={"ID":"be5a7a49-6ba1-410f-91bc-fa13983a7b94","Type":"ContainerStarted","Data":"17138d5ae8a4b4f889cd79a5ac1a1c51239d39b924138b9900a440150e1ca10a"}
Jul 30 17:00:56.761270 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.761235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" event={"ID":"81aff41b649dded9cc1db78fddd9827a","Type":"ContainerStarted","Data":"0de9dc140f8449cb3e2a10ecc6143c77ef1ab36c25575b652c3cb7a5a2eb7894"}
Jul 30 17:00:56.777600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:56.777548    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-31.ec2.internal" podStartSLOduration=22.777527115 podStartE2EDuration="22.777527115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:56.777302538 +0000 UTC m=+24.810112877" watchObservedRunningTime="2026-07-30 17:00:56.777527115 +0000 UTC m=+24.810337456"
Jul 30 17:00:57.646673 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:57.646582    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:57.646837 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:57.646696    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:00:57.768461 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:57.768423    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"62b7510fd29322c350ae9160780a67400779c03a3c10414f157f60830575be76"}
Jul 30 17:00:57.770896 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:57.770852    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg" event={"ID":"be5a7a49-6ba1-410f-91bc-fa13983a7b94","Type":"ContainerStarted","Data":"1d369d20ffdc3e56f55afbd71a7ad45756b6abfaa463a4e879154cc77f7f7ba9"}
Jul 30 17:00:57.771047 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:57.770908    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 17:00:58.647636 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:58.647608    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:00:58.647842 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:58.647714    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:00:58.647842 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:58.647769    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:00:58.647984 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:58.647865    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:00:59.646797 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:00:59.646764    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:00:59.647199 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:00:59.646880    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:00.647403 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.647113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:00.648000 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:00.647482    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:00.648000 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.647160    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:00.648000 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:00.647599    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:00.781789 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.781761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" event={"ID":"42f582b9-f05b-4c3e-996f-eebb4118a05a","Type":"ContainerStarted","Data":"75c09908c6ad69c38e7998dec52017f776fae1c2911d2eea9e67ae78969e5eb8"}
Jul 30 17:01:00.782390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.782182    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:01:00.782390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.782282    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:01:00.782390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.782333    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:01:00.783842 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.783816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerStarted","Data":"3c19ebbc07cb86cf5b2577d13e2b44392d3269968c200bb31461090c6a9d23a4"}
Jul 30 17:01:00.797170 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.797151    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:01:00.797301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.797245    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:01:00.809602 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.809565    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cswfg" podStartSLOduration=10.338756978 podStartE2EDuration="27.809552051s" totalImagesPullingTime="17.470795073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:00:57.789782121 +0000 UTC m=+25.822592458" watchObservedRunningTime="2026-07-30 17:01:00.809552051 +0000 UTC m=+28.842362428"
Jul 30 17:01:00.809740 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:00.809721    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99" podStartSLOduration=8.075851533 podStartE2EDuration="27.809715879s" totalImagesPullingTime="19.733864346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:00.809052756 +0000 UTC m=+28.841863104" watchObservedRunningTime="2026-07-30 17:01:00.809715879 +0000 UTC m=+28.842526215"
Jul 30 17:01:01.647205 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:01.647169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:01.647395 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:01.647293    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:01.788117 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:01.788080    2569 generic.go:356] "Generic (PLEG): container finished" podID="072815d7-8920-4d28-9e01-95e3a9908d71" containerID="3c19ebbc07cb86cf5b2577d13e2b44392d3269968c200bb31461090c6a9d23a4" exitCode=0
Jul 30 17:01:01.788747 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:01.788164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerDied","Data":"3c19ebbc07cb86cf5b2577d13e2b44392d3269968c200bb31461090c6a9d23a4"}
Jul 30 17:01:01.926634 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:01.926553    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:01:01.926784 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:01.926668    2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 17:01:01.927092 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:01.927079    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sjw84"
Jul 30 17:01:02.652886 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:02.652849    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:02.653069 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:02.653029    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:02.653334 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:02.653310    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:02.653836 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:02.653801    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:03.647414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:03.647374    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:03.647899 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:03.647483    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:03.793895 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:03.793860    2569 generic.go:356] "Generic (PLEG): container finished" podID="072815d7-8920-4d28-9e01-95e3a9908d71" containerID="06a12258443326a5d084202917e884cf10e4c4f12691940e3ed5ee261b6ebb28" exitCode=0
Jul 30 17:01:03.794056 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:03.793902    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerDied","Data":"06a12258443326a5d084202917e884cf10e4c4f12691940e3ed5ee261b6ebb28"}
Jul 30 17:01:04.646883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:04.646797    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:04.647029 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:04.646898    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:04.647029 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:04.646944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:04.647029 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:04.646999    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:04.797803 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:04.797773    2569 generic.go:356] "Generic (PLEG): container finished" podID="072815d7-8920-4d28-9e01-95e3a9908d71" containerID="183b97a35040363ead337cadfbd4e1f2847ea451940201dd360e9b3b1dd09342" exitCode=0
Jul 30 17:01:04.798174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:04.797814    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerDied","Data":"183b97a35040363ead337cadfbd4e1f2847ea451940201dd360e9b3b1dd09342"}
Jul 30 17:01:05.646795 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:05.646763    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:05.646958 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:05.646866    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:06.160653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:06.160614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:06.161221 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.160743    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 17:01:06.161221 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.160809    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret podName:5962bc8b-6237-4100-830f-5a54e63e0f14 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.160789283 +0000 UTC m=+66.193599604 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret") pod "global-pull-secret-syncer-2w6gg" (UID: "5962bc8b-6237-4100-830f-5a54e63e0f14") : object "kube-system"/"original-pull-secret" not registered
Jul 30 17:01:06.261155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:06.261114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:06.261346 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:06.261173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:06.261346 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.261295    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:01:06.261346 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.261300    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 17:01:06.261346 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.261330    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 17:01:06.261346 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.261344    2569 projected.go:196] Error preparing data for projected volume kube-api-access-ml4sq for pod openshift-network-diagnostics/network-check-target-5gzq2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:01:06.261564 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.261362    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs podName:ffa93763-d018-4c80-aa34-83bc592bc8b8 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.261342225 +0000 UTC m=+66.294152543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs") pod "network-metrics-daemon-qfh2d" (UID: "ffa93763-d018-4c80-aa34-83bc592bc8b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 17:01:06.261564 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.261395    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq podName:9587b1cf-a314-4e21-b752-1cadd4877321 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:38.261378821 +0000 UTC m=+66.294189153 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ml4sq" (UniqueName: "kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq") pod "network-check-target-5gzq2" (UID: "9587b1cf-a314-4e21-b752-1cadd4877321") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 17:01:06.650148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:06.650119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:06.650349 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.650244    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:06.650349 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:06.650305    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:06.650467 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:06.650431    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:07.647990 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:07.647458    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:07.647990 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:07.647573    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.651653    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gzq2"]
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.651916    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:08.652016    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.652172    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:08.652227    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.652511    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2w6gg"]
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.652537    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:08.652689    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:08.653701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.653638    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qfh2d"]
Jul 30 17:01:08.810477 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.808144    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:08.810477 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:08.808346    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:08.810477 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:08.808141    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:08.810477 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:08.808738    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:09.812737 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:09.812704    2569 generic.go:356] "Generic (PLEG): container finished" podID="072815d7-8920-4d28-9e01-95e3a9908d71" containerID="6947b871491f8d055202971662870c346098b7b5b4e6a1d12556b450556ce4eb" exitCode=0
Jul 30 17:01:09.813157 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:09.812751    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerDied","Data":"6947b871491f8d055202971662870c346098b7b5b4e6a1d12556b450556ce4eb"}
Jul 30 17:01:10.647353 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:10.647319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:10.647524 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:10.647319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:10.647524 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:10.647452    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qfh2d" podUID="ffa93763-d018-4c80-aa34-83bc592bc8b8"
Jul 30 17:01:10.647645 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:10.647539    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2w6gg" podUID="5962bc8b-6237-4100-830f-5a54e63e0f14"
Jul 30 17:01:10.647645 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:10.647324    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:10.647645 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:10.647632    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5gzq2" podUID="9587b1cf-a314-4e21-b752-1cadd4877321"
Jul 30 17:01:10.817380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:10.817306    2569 generic.go:356] "Generic (PLEG): container finished" podID="072815d7-8920-4d28-9e01-95e3a9908d71" containerID="c96fe82cae0bef7787f452f9a213a6588e96c68b898e10e7459108801df02364" exitCode=0
Jul 30 17:01:10.817380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:10.817348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerDied","Data":"c96fe82cae0bef7787f452f9a213a6588e96c68b898e10e7459108801df02364"}
Jul 30 17:01:11.822811 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:11.822621    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-swrqn" event={"ID":"072815d7-8920-4d28-9e01-95e3a9908d71","Type":"ContainerStarted","Data":"61e40b37bf4e284b15bb03eed9e5aca2ddc68297f2068a29adddc2c20cd06cdc"}
Jul 30 17:01:11.846165 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:11.846116    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-swrqn" podStartSLOduration=9.06051552 podStartE2EDuration="38.846103539s" totalImagesPullingTime="29.785588019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:11.845893121 +0000 UTC m=+39.878703457" watchObservedRunningTime="2026-07-30 17:01:11.846103539 +0000 UTC m=+39.878913875"
Jul 30 17:01:12.212033 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.212004    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-31.ec2.internal" event="NodeReady"
Jul 30 17:01:12.212201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.212116    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 17:01:12.243679 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.243650    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d6d5bb4cc-566rf"]
Jul 30 17:01:12.267814 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.267789    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d6d5bb4cc-566rf"]
Jul 30 17:01:12.267814 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.267818    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8k7v2"]
Jul 30 17:01:12.268010 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.267923    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.270141 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.270120    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 30 17:01:12.270839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.270808    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 30 17:01:12.270951 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.270815    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8q6mn\""
Jul 30 17:01:12.270951 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.270818    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 30 17:01:12.276007 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.275442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 30 17:01:12.286215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.286198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.286215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.286206    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9rcbw"]
Jul 30 17:01:12.289197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.289178    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 17:01:12.289197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.289196    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 17:01:12.289336 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.289307    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c76qr\""
Jul 30 17:01:12.289563 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.289548    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 17:01:12.306328 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.306305    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8k7v2"]
Jul 30 17:01:12.306414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.306333    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9rcbw"]
Jul 30 17:01:12.306461 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.306412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.309060 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.309044    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 17:01:12.309143 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.309082    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 17:01:12.309424 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.309409    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k8vp7\""
Jul 30 17:01:12.309480 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.309430    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 17:01:12.309480 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.309471    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 17:01:12.359645 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.359620    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-689qd"]
Jul 30 17:01:12.380867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.380846    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-689qd"]
Jul 30 17:01:12.380966 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.380926    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.383581 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.383565    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 17:01:12.383768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.383754    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 17:01:12.383861 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.383848    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hvt8c\""
Jul 30 17:01:12.406822 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406803    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rv77\" (UniqueName: \"kubernetes.io/projected/5964c201-0d3d-444d-b808-33a3aaf5522e-kube-api-access-2rv77\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.406937 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2906046b-8db0-4b67-a26d-5dcaeaaeb057-trusted-ca\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.406937 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5964c201-0d3d-444d-b808-33a3aaf5522e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.406937 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406898    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2906046b-8db0-4b67-a26d-5dcaeaaeb057-registry-certificates\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwcn\" (UniqueName: \"kubernetes.io/projected/d476091d-f0e7-48d1-a149-f4bf94cf817b-kube-api-access-2fwcn\") pod \"ingress-canary-8k7v2\" (UID: \"d476091d-f0e7-48d1-a149-f4bf94cf817b\") " pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5964c201-0d3d-444d-b808-33a3aaf5522e-crio-socket\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406981    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5964c201-0d3d-444d-b808-33a3aaf5522e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.406997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f947g\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-kube-api-access-f947g\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407013    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-registry-tls\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d476091d-f0e7-48d1-a149-f4bf94cf817b-cert\") pod \"ingress-canary-8k7v2\" (UID: \"d476091d-f0e7-48d1-a149-f4bf94cf817b\") " pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.407076 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2906046b-8db0-4b67-a26d-5dcaeaaeb057-ca-trust-extracted\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.407414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2906046b-8db0-4b67-a26d-5dcaeaaeb057-image-registry-private-configuration\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.407414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407101    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5964c201-0d3d-444d-b808-33a3aaf5522e-data-volume\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.407414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407142    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2906046b-8db0-4b67-a26d-5dcaeaaeb057-installation-pull-secrets\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.407414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.407169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-bound-sa-token\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507489 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnjlf\" (UniqueName: \"kubernetes.io/projected/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-kube-api-access-tnjlf\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.507489 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwcn\" (UniqueName: \"kubernetes.io/projected/d476091d-f0e7-48d1-a149-f4bf94cf817b-kube-api-access-2fwcn\") pod \"ingress-canary-8k7v2\" (UID: \"d476091d-f0e7-48d1-a149-f4bf94cf817b\") " pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.507674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507489    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5964c201-0d3d-444d-b808-33a3aaf5522e-crio-socket\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.507674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5964c201-0d3d-444d-b808-33a3aaf5522e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.507674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507596    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f947g\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-kube-api-access-f947g\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507615    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5964c201-0d3d-444d-b808-33a3aaf5522e-crio-socket\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.507674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-registry-tls\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507652    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-tmp-dir\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d476091d-f0e7-48d1-a149-f4bf94cf817b-cert\") pod \"ingress-canary-8k7v2\" (UID: \"d476091d-f0e7-48d1-a149-f4bf94cf817b\") " pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2906046b-8db0-4b67-a26d-5dcaeaaeb057-ca-trust-extracted\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507788    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-metrics-tls\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507814    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2906046b-8db0-4b67-a26d-5dcaeaaeb057-image-registry-private-configuration\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5964c201-0d3d-444d-b808-33a3aaf5522e-data-volume\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2906046b-8db0-4b67-a26d-5dcaeaaeb057-installation-pull-secrets\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-bound-sa-token\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rv77\" (UniqueName: \"kubernetes.io/projected/5964c201-0d3d-444d-b808-33a3aaf5522e-kube-api-access-2rv77\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2906046b-8db0-4b67-a26d-5dcaeaaeb057-trusted-ca\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.507985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.507989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5964c201-0d3d-444d-b808-33a3aaf5522e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.508472 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.508014    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2906046b-8db0-4b67-a26d-5dcaeaaeb057-registry-certificates\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.508472 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.508039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-config-volume\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.508472 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.508179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2906046b-8db0-4b67-a26d-5dcaeaaeb057-ca-trust-extracted\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.508472 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.508330    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5964c201-0d3d-444d-b808-33a3aaf5522e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.508971 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.508718    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5964c201-0d3d-444d-b808-33a3aaf5522e-data-volume\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.509395 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.509343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2906046b-8db0-4b67-a26d-5dcaeaaeb057-trusted-ca\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.509726 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.509701    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2906046b-8db0-4b67-a26d-5dcaeaaeb057-registry-certificates\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.512448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.512349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2906046b-8db0-4b67-a26d-5dcaeaaeb057-installation-pull-secrets\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.512448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.512351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5964c201-0d3d-444d-b808-33a3aaf5522e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.512448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.512417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2906046b-8db0-4b67-a26d-5dcaeaaeb057-image-registry-private-configuration\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.512883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.512859    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-registry-tls\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.512967 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.512893    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d476091d-f0e7-48d1-a149-f4bf94cf817b-cert\") pod \"ingress-canary-8k7v2\" (UID: \"d476091d-f0e7-48d1-a149-f4bf94cf817b\") " pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.516987 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.516959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-bound-sa-token\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.517451 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.517431    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwcn\" (UniqueName: \"kubernetes.io/projected/d476091d-f0e7-48d1-a149-f4bf94cf817b-kube-api-access-2fwcn\") pod \"ingress-canary-8k7v2\" (UID: \"d476091d-f0e7-48d1-a149-f4bf94cf817b\") " pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.517583 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.517561    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rv77\" (UniqueName: \"kubernetes.io/projected/5964c201-0d3d-444d-b808-33a3aaf5522e-kube-api-access-2rv77\") pod \"insights-runtime-extractor-9rcbw\" (UID: \"5964c201-0d3d-444d-b808-33a3aaf5522e\") " pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.518174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.518154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f947g\" (UniqueName: \"kubernetes.io/projected/2906046b-8db0-4b67-a26d-5dcaeaaeb057-kube-api-access-f947g\") pod \"image-registry-d6d5bb4cc-566rf\" (UID: \"2906046b-8db0-4b67-a26d-5dcaeaaeb057\") " pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.578705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.578675    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:12.594800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.594776    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8k7v2"
Jul 30 17:01:12.609343 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.609313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-config-volume\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.609507 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.609346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnjlf\" (UniqueName: \"kubernetes.io/projected/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-kube-api-access-tnjlf\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.609507 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.609390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-tmp-dir\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.609507 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.609418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-metrics-tls\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.609759 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.609736    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-tmp-dir\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.609903 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.609888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-config-volume\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.611625 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.611602    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-metrics-tls\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.615552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.615527    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9rcbw"
Jul 30 17:01:12.618686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.618663    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnjlf\" (UniqueName: \"kubernetes.io/projected/0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299-kube-api-access-tnjlf\") pod \"dns-default-689qd\" (UID: \"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299\") " pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.647943 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.647918    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:12.648063 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.648036    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:12.648423 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.648198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:12.651416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.651221    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 17:01:12.651416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.651230    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 17:01:12.651416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.651307    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 17:01:12.651416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.651359    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4wpnm\""
Jul 30 17:01:12.651705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.651504    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 17:01:12.651705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.651615    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wcszl\""
Jul 30 17:01:12.691411 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.691383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:12.785635 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.784913    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8k7v2"]
Jul 30 17:01:12.788423 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:12.788368    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd476091d_f0e7_48d1_a149_f4bf94cf817b.slice/crio-f05ddb2c13fae294bd13a6f0940f70dfbec6e6f249d2ba9662c83a559ae88b27 WatchSource:0}: Error finding container f05ddb2c13fae294bd13a6f0940f70dfbec6e6f249d2ba9662c83a559ae88b27: Status 404 returned error can't find the container with id f05ddb2c13fae294bd13a6f0940f70dfbec6e6f249d2ba9662c83a559ae88b27
Jul 30 17:01:12.809012 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.808986    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d6d5bb4cc-566rf"]
Jul 30 17:01:12.815002 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:12.814974    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2906046b_8db0_4b67_a26d_5dcaeaaeb057.slice/crio-81fc9866492dc280ed4094a40389e3455b56f648ae4cf4ef4091046f58f49c62 WatchSource:0}: Error finding container 81fc9866492dc280ed4094a40389e3455b56f648ae4cf4ef4091046f58f49c62: Status 404 returned error can't find the container with id 81fc9866492dc280ed4094a40389e3455b56f648ae4cf4ef4091046f58f49c62
Jul 30 17:01:12.815180 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.815158    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9rcbw"]
Jul 30 17:01:12.818898 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:12.818813    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5964c201_0d3d_444d_b808_33a3aaf5522e.slice/crio-1c5580ea2ea31dd386ea4bf7f06b59929d7868e837133f1c569602a0beb39328 WatchSource:0}: Error finding container 1c5580ea2ea31dd386ea4bf7f06b59929d7868e837133f1c569602a0beb39328: Status 404 returned error can't find the container with id 1c5580ea2ea31dd386ea4bf7f06b59929d7868e837133f1c569602a0beb39328
Jul 30 17:01:12.826723 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.826698    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf" event={"ID":"2906046b-8db0-4b67-a26d-5dcaeaaeb057","Type":"ContainerStarted","Data":"81fc9866492dc280ed4094a40389e3455b56f648ae4cf4ef4091046f58f49c62"}
Jul 30 17:01:12.827910 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.827883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rcbw" event={"ID":"5964c201-0d3d-444d-b808-33a3aaf5522e","Type":"ContainerStarted","Data":"1c5580ea2ea31dd386ea4bf7f06b59929d7868e837133f1c569602a0beb39328"}
Jul 30 17:01:12.828946 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.828919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8k7v2" event={"ID":"d476091d-f0e7-48d1-a149-f4bf94cf817b","Type":"ContainerStarted","Data":"f05ddb2c13fae294bd13a6f0940f70dfbec6e6f249d2ba9662c83a559ae88b27"}
Jul 30 17:01:12.862648 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:12.862622    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-689qd"]
Jul 30 17:01:12.867185 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:12.867158    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7a4eed_fed3_4c08_9fb3_4f1ae6a4d299.slice/crio-535e637b40af37ad3e48e6cd58d3dc7990f1e99481a8337ab1b45d848c84fb9f WatchSource:0}: Error finding container 535e637b40af37ad3e48e6cd58d3dc7990f1e99481a8337ab1b45d848c84fb9f: Status 404 returned error can't find the container with id 535e637b40af37ad3e48e6cd58d3dc7990f1e99481a8337ab1b45d848c84fb9f
Jul 30 17:01:13.833670 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:13.833631    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-689qd" event={"ID":"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299","Type":"ContainerStarted","Data":"535e637b40af37ad3e48e6cd58d3dc7990f1e99481a8337ab1b45d848c84fb9f"}
Jul 30 17:01:13.835634 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:13.835598    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf" event={"ID":"2906046b-8db0-4b67-a26d-5dcaeaaeb057","Type":"ContainerStarted","Data":"c4baf7f9ed68c58c8f8d3b9370862026b9e08249b77056abbec5d3a01c9d96db"}
Jul 30 17:01:13.835803 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:13.835784    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:13.857110 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:13.857061    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf" podStartSLOduration=6.857044258 podStartE2EDuration="6.857044258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:13.855895902 +0000 UTC m=+41.888706241" watchObservedRunningTime="2026-07-30 17:01:13.857044258 +0000 UTC m=+41.889854588"
Jul 30 17:01:16.845498 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:16.845456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8k7v2" event={"ID":"d476091d-f0e7-48d1-a149-f4bf94cf817b","Type":"ContainerStarted","Data":"33bb11a23ed18da1dc840cea263da690851f1b2a6d40a87620d09aa5dcb705f0"}
Jul 30 17:01:16.847505 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:16.847481    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-689qd" event={"ID":"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299","Type":"ContainerStarted","Data":"855263e0d179e4c9b037a75b8589deddb4688bfb50f2eadb435a5376e2c65b68"}
Jul 30 17:01:16.847640 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:16.847521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-689qd" event={"ID":"0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299","Type":"ContainerStarted","Data":"f4a003eb0c3fa5eee3617cb2c248158e92ebb826ba52ff0f8cd232ab1f2fbf7c"}
Jul 30 17:01:16.847706 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:16.847652    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:16.862336 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:16.862287    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8k7v2" podStartSLOduration=1.497979103 podStartE2EDuration="4.862271863s" totalImagesPullingTime="3.36429276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:16.862068049 +0000 UTC m=+44.894878388" watchObservedRunningTime="2026-07-30 17:01:16.862271863 +0000 UTC m=+44.895082193"
Jul 30 17:01:16.878820 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:16.878771    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-689qd" podStartSLOduration=1.587373425 podStartE2EDuration="4.878753532s" totalImagesPullingTime="3.291380107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:16.877954124 +0000 UTC m=+44.910764463" watchObservedRunningTime="2026-07-30 17:01:16.878753532 +0000 UTC m=+44.911563869"
Jul 30 17:01:19.259133 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.259069    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"]
Jul 30 17:01:19.427250 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.427213    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"]
Jul 30 17:01:19.427440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.427352    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:19.430065 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.430036    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 30 17:01:19.430191 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.430079    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vfzcf\""
Jul 30 17:01:19.460284 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.460243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/00ae6fa6-b8e2-43ec-aba5-d436a0aedb48-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ghqtm\" (UID: \"00ae6fa6-b8e2-43ec-aba5-d436a0aedb48\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:19.561171 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.561083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/00ae6fa6-b8e2-43ec-aba5-d436a0aedb48-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ghqtm\" (UID: \"00ae6fa6-b8e2-43ec-aba5-d436a0aedb48\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:19.563880 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.563849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/00ae6fa6-b8e2-43ec-aba5-d436a0aedb48-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-ghqtm\" (UID: \"00ae6fa6-b8e2-43ec-aba5-d436a0aedb48\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:19.738798 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:19.738759    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:21.036990 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:21.036949    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"]
Jul 30 17:01:21.040154 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:21.040127    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ae6fa6_b8e2_43ec_aba5_d436a0aedb48.slice/crio-fee84d31d2907c548e3629c9132ef794ef89e33c57acce99fb55d81c5178e50a WatchSource:0}: Error finding container fee84d31d2907c548e3629c9132ef794ef89e33c57acce99fb55d81c5178e50a: Status 404 returned error can't find the container with id fee84d31d2907c548e3629c9132ef794ef89e33c57acce99fb55d81c5178e50a
Jul 30 17:01:21.862471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:21.862431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rcbw" event={"ID":"5964c201-0d3d-444d-b808-33a3aaf5522e","Type":"ContainerStarted","Data":"13717c4ce1f52aef497dec7cf39cee264e2d9c6578c7eb72019b5306e40b7046"}
Jul 30 17:01:21.863687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:21.863653    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm" event={"ID":"00ae6fa6-b8e2-43ec-aba5-d436a0aedb48","Type":"ContainerStarted","Data":"fee84d31d2907c548e3629c9132ef794ef89e33c57acce99fb55d81c5178e50a"}
Jul 30 17:01:23.872517 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:23.872469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm" event={"ID":"00ae6fa6-b8e2-43ec-aba5-d436a0aedb48","Type":"ContainerStarted","Data":"5dffbe225f07af6ffc2b14badb80749ef53400f0219c52ca0bc73194d35b4989"}
Jul 30 17:01:23.872973 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:23.872866    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:23.878867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:23.878841    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm"
Jul 30 17:01:23.889186 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:23.889138    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-ghqtm" podStartSLOduration=3.125511102 podStartE2EDuration="4.889122933s" totalImagesPullingTime="1.763611831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:23.888235918 +0000 UTC m=+51.921046256" watchObservedRunningTime="2026-07-30 17:01:23.889122933 +0000 UTC m=+51.921933270"
Jul 30 17:01:25.882696 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:25.882666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rcbw" event={"ID":"5964c201-0d3d-444d-b808-33a3aaf5522e","Type":"ContainerStarted","Data":"9af4bec49fcebb95ff2cc0fefdc22db41d4d16c671f284fa181f4f01d2708000"}
Jul 30 17:01:26.852749 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:26.852718    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-689qd"
Jul 30 17:01:27.341206 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.341167    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dd4dfffbf-bwkd9"]
Jul 30 17:01:27.343416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.343396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.347429 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.347400    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 30 17:01:27.347429 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.347416    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 30 17:01:27.347602 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.347429    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 30 17:01:27.348027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.347840    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 30 17:01:27.348027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.347888    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 30 17:01:27.348027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.347914    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 30 17:01:27.348247 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.348154    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5rx9s\""
Jul 30 17:01:27.348247 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.348183    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 30 17:01:27.355859 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.355835    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd4dfffbf-bwkd9"]
Jul 30 17:01:27.419170 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.419134    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-oauth-config\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.419170 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.419172    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-serving-cert\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.419441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.419200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-config\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.419441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.419371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-service-ca\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.419441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.419408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-oauth-serving-cert\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.419571 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.419440    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqfg\" (UniqueName: \"kubernetes.io/projected/358c72ae-b819-4dee-a147-f94e1e6a9f6a-kube-api-access-rlqfg\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.520363 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.520327    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-service-ca\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.520363 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.520367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-oauth-serving-cert\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.520600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.520395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqfg\" (UniqueName: \"kubernetes.io/projected/358c72ae-b819-4dee-a147-f94e1e6a9f6a-kube-api-access-rlqfg\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.520600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.520449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-oauth-config\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.520600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.520477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-serving-cert\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.520600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.520498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-config\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.521220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.521166    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-oauth-serving-cert\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.521220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.521176    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-service-ca\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.521220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.521218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-config\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.523229 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.523179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-oauth-config\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.523364 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.523328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-serving-cert\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.530136 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.530107    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqfg\" (UniqueName: \"kubernetes.io/projected/358c72ae-b819-4dee-a147-f94e1e6a9f6a-kube-api-access-rlqfg\") pod \"console-dd4dfffbf-bwkd9\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") " pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:27.654746 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:27.654654    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:29.306289 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:29.306249    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dd4dfffbf-bwkd9"]
Jul 30 17:01:29.309604 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:29.309571    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358c72ae_b819_4dee_a147_f94e1e6a9f6a.slice/crio-719a84ce33647055236c773d0d0b1372c2d63f10c40d55e638af07124cd49230 WatchSource:0}: Error finding container 719a84ce33647055236c773d0d0b1372c2d63f10c40d55e638af07124cd49230: Status 404 returned error can't find the container with id 719a84ce33647055236c773d0d0b1372c2d63f10c40d55e638af07124cd49230
Jul 30 17:01:29.900643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:29.900295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9rcbw" event={"ID":"5964c201-0d3d-444d-b808-33a3aaf5522e","Type":"ContainerStarted","Data":"a4e803ba73d8079aa380a3221a3cfdaaf5521e98683442053cc26151bd4e8a7d"}
Jul 30 17:01:29.901710 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:29.901677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4dfffbf-bwkd9" event={"ID":"358c72ae-b819-4dee-a147-f94e1e6a9f6a","Type":"ContainerStarted","Data":"719a84ce33647055236c773d0d0b1372c2d63f10c40d55e638af07124cd49230"}
Jul 30 17:01:29.919652 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:29.919600    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9rcbw" podStartSLOduration=1.6931917699999999 podStartE2EDuration="17.919583552s" totalImagesPullingTime="16.226391782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:01:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:29.918825313 +0000 UTC m=+57.951635651" watchObservedRunningTime="2026-07-30 17:01:29.919583552 +0000 UTC m=+57.952393891"
Jul 30 17:01:31.732548 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.732514    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hzkvn"]
Jul 30 17:01:31.750561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.750533    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"]
Jul 30 17:01:31.750737 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.750676    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.753226 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.753204    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 17:01:31.753389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.753323    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 17:01:31.753483 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.753459    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 17:01:31.753739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.753703    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 17:01:31.754617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.754543    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 17:01:31.754617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.754544    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hq64z\""
Jul 30 17:01:31.754617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.754544    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 17:01:31.772647 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.772618    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"]
Jul 30 17:01:31.772772 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.772759    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.775408 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.775384    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 30 17:01:31.775580 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.775556    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 30 17:01:31.775959 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.775929    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 30 17:01:31.776059 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.775976    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qp6px\""
Jul 30 17:01:31.860834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.860793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.861009 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.860852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-accelerators-collector-config\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861009 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.860964    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-sys\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861009 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.860995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-wtmp\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.861154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861055    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-root\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-tls\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861132    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22db70d2-fa87-4b62-99ff-38988b3e2e63-metrics-client-ca\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861345 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861170    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-textfile\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861345 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tjd\" (UniqueName: \"kubernetes.io/projected/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-api-access-54tjd\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.861417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.861417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87r2\" (UniqueName: \"kubernetes.io/projected/22db70d2-fa87-4b62-99ff-38988b3e2e63-kube-api-access-z87r2\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.861541 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.861541 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.861449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tjd\" (UniqueName: \"kubernetes.io/projected/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-api-access-54tjd\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962595    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87r2\" (UniqueName: \"kubernetes.io/projected/22db70d2-fa87-4b62-99ff-38988b3e2e63-kube-api-access-z87r2\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:31.962758    2569 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962760    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.962834 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:31.962841    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-tls podName:b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:32.462807605 +0000 UTC m=+60.495617930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-dknvj" (UID: "b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01") : secret "kube-state-metrics-tls" not found
Jul 30 17:01:31.963484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.963484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.963484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.962971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-accelerators-collector-config\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.963484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.963024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-sys\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.963484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.963056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-wtmp\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.964090 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.964181 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.964181 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964145    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-root\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.964181 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-tls\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.964365 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964224    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22db70d2-fa87-4b62-99ff-38988b3e2e63-metrics-client-ca\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.964365 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-textfile\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964604    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-textfile\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.964750    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-wtmp\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.965222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-accelerators-collector-config\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.965249    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-root\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.965297    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22db70d2-fa87-4b62-99ff-38988b3e2e63-sys\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.965472    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:31.965559    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:31.965620    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-tls podName:22db70d2-fa87-4b62-99ff-38988b3e2e63 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:32.465603002 +0000 UTC m=+60.498413324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-tls") pod "node-exporter-hzkvn" (UID: "22db70d2-fa87-4b62-99ff-38988b3e2e63") : secret "node-exporter-tls" not found
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.965557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.966035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22db70d2-fa87-4b62-99ff-38988b3e2e63-metrics-client-ca\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.966093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.966063    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.967863 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.967821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:31.971865 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.971842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87r2\" (UniqueName: \"kubernetes.io/projected/22db70d2-fa87-4b62-99ff-38988b3e2e63-kube-api-access-z87r2\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:31.971865 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:31.971858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tjd\" (UniqueName: \"kubernetes.io/projected/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-api-access-54tjd\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:32.467084 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.467021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:32.467248 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.467133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-tls\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:32.469530 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.469501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22db70d2-fa87-4b62-99ff-38988b3e2e63-node-exporter-tls\") pod \"node-exporter-hzkvn\" (UID: \"22db70d2-fa87-4b62-99ff-38988b3e2e63\") " pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:32.469656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.469563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dknvj\" (UID: \"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:32.583080 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.583049    2569 patch_prober.go:28] interesting pod/image-registry-d6d5bb4cc-566rf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 30 17:01:32.583236 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.583099    2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf" podUID="2906046b-8db0-4b67-a26d-5dcaeaaeb057" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:01:32.664164 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.664136    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hq64z\""
Jul 30 17:01:32.672471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.672445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hzkvn"
Jul 30 17:01:32.686319 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.686294    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qp6px\""
Jul 30 17:01:32.694278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.694244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"
Jul 30 17:01:32.805732 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.805667    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9f99"
Jul 30 17:01:32.807712 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.807690    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:01:32.822953 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.822930    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.825996 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.825970    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 30 17:01:32.825996 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.825994    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 30 17:01:32.826232 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826097    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 30 17:01:32.826232 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.825983    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 30 17:01:32.826232 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826165    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-62qft\""
Jul 30 17:01:32.826379 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826245    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 30 17:01:32.826426 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826397    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 30 17:01:32.826478 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826426    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 30 17:01:32.826478 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826445    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 30 17:01:32.826478 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.826465    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 30 17:01:32.830737 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.830715    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:01:32.869546 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869518    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.869718 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.869718 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.869718 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.869912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.869912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870021 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm62g\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-kube-api-access-hm62g\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870021 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.869949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870119 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.870021    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-out\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870119 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.870051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870119 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.870088    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-web-config\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870300 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.870143    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.870300 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.870186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.877504 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:32.877476    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22db70d2_fa87_4b62_99ff_38988b3e2e63.slice/crio-106b5822888b9a7f5a9adf647595ef6df7a5fc9ddcb37857eddf4c1109c0b049 WatchSource:0}: Error finding container 106b5822888b9a7f5a9adf647595ef6df7a5fc9ddcb37857eddf4c1109c0b049: Status 404 returned error can't find the container with id 106b5822888b9a7f5a9adf647595ef6df7a5fc9ddcb37857eddf4c1109c0b049
Jul 30 17:01:32.914321 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.914286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzkvn" event={"ID":"22db70d2-fa87-4b62-99ff-38988b3e2e63","Type":"ContainerStarted","Data":"106b5822888b9a7f5a9adf647595ef6df7a5fc9ddcb37857eddf4c1109c0b049"}
Jul 30 17:01:32.970936 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.970909    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.970950    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.970979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm62g\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-kube-api-access-hm62g\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.970999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-out\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-web-config\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971202    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971550 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971550 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971550 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.971407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.971696 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:01:32.971659    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle podName:8fbe0826-f5fa-402b-9d1f-8f6c13c79709 nodeName:}" failed. No retries permitted until 2026-07-30 17:01:33.47162139 +0000 UTC m=+61.504431724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709") : configmap references non-existent config key: ca-bundle.crt
Jul 30 17:01:32.975421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.975235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.975796 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.975771    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-web-config\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.976245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.976223    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.979668 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.977620    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.979668 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.977895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.979668 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.978412    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-out\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.979668 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.978446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.982767 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.982479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.983011 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.982895    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.984197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.984174    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.984818 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.984789    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm62g\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-kube-api-access-hm62g\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:32.995704 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:32.995677    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj"]
Jul 30 17:01:33.000056 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:33.000031    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17d84c4_fdb6_4fda_8e7d_ddaef1e5fd01.slice/crio-1600817197f938c3d77f5c9441f8c6910c4d0d9f719ce7e1b49dc5be9284d2e7 WatchSource:0}: Error finding container 1600817197f938c3d77f5c9441f8c6910c4d0d9f719ce7e1b49dc5be9284d2e7: Status 404 returned error can't find the container with id 1600817197f938c3d77f5c9441f8c6910c4d0d9f719ce7e1b49dc5be9284d2e7
Jul 30 17:01:33.475366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.475337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:33.476034 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.476014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:33.732410 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.732319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:01:33.879607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.879570    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:01:33.884100 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:33.884072    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbe0826_f5fa_402b_9d1f_8f6c13c79709.slice/crio-27b0d6e3b7932d1fb27c889fc389936d49c6fc43b33d9f94b35cec2293bcacf4 WatchSource:0}: Error finding container 27b0d6e3b7932d1fb27c889fc389936d49c6fc43b33d9f94b35cec2293bcacf4: Status 404 returned error can't find the container with id 27b0d6e3b7932d1fb27c889fc389936d49c6fc43b33d9f94b35cec2293bcacf4
Jul 30 17:01:33.920338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.920299    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4dfffbf-bwkd9" event={"ID":"358c72ae-b819-4dee-a147-f94e1e6a9f6a","Type":"ContainerStarted","Data":"ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f"}
Jul 30 17:01:33.921724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.921687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj" event={"ID":"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01","Type":"ContainerStarted","Data":"1600817197f938c3d77f5c9441f8c6910c4d0d9f719ce7e1b49dc5be9284d2e7"}
Jul 30 17:01:33.922968 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.922936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"27b0d6e3b7932d1fb27c889fc389936d49c6fc43b33d9f94b35cec2293bcacf4"}
Jul 30 17:01:33.938889 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:33.938839    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dd4dfffbf-bwkd9" podStartSLOduration=3.372604079 podStartE2EDuration="6.938825457s" totalImagesPullingTime="3.566221378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:33.937249513 +0000 UTC m=+61.970059849" watchObservedRunningTime="2026-07-30 17:01:33.938825457 +0000 UTC m=+61.971635795"
Jul 30 17:01:34.843117 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:34.843089    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d6d5bb4cc-566rf"
Jul 30 17:01:34.928611 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:34.928569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzkvn" event={"ID":"22db70d2-fa87-4b62-99ff-38988b3e2e63","Type":"ContainerStarted","Data":"b46642bc2a4e0184d130b560cabbd7b05f2e4a268aa38af02480dd0b623f9fa9"}
Jul 30 17:01:34.931024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:34.930992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj" event={"ID":"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01","Type":"ContainerStarted","Data":"c30f5f04eec84389c08f56e8a723c2b3493e17fd2805dbaa641a81d4abe883dc"}
Jul 30 17:01:34.931148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:34.931063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj" event={"ID":"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01","Type":"ContainerStarted","Data":"c452e6e5c82c290f207610e67e642f708282c8745972e63145c886ff8384bd89"}
Jul 30 17:01:34.931148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:34.931099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj" event={"ID":"b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01","Type":"ContainerStarted","Data":"6a81871cad9ccbb6a867a8c723a7feb13734b1aa8c15fda72bd98388d564286b"}
Jul 30 17:01:34.969743 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:34.969685    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dknvj" podStartSLOduration=2.60276579 podStartE2EDuration="3.969665565s" totalImagesPullingTime="1.366899775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:34.968671064 +0000 UTC m=+63.001481399" watchObservedRunningTime="2026-07-30 17:01:34.969665565 +0000 UTC m=+63.002475903"
Jul 30 17:01:35.936471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:35.936430    2569 generic.go:356] "Generic (PLEG): container finished" podID="22db70d2-fa87-4b62-99ff-38988b3e2e63" containerID="b46642bc2a4e0184d130b560cabbd7b05f2e4a268aa38af02480dd0b623f9fa9" exitCode=0
Jul 30 17:01:35.936905 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:35.936469    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzkvn" event={"ID":"22db70d2-fa87-4b62-99ff-38988b3e2e63","Type":"ContainerDied","Data":"b46642bc2a4e0184d130b560cabbd7b05f2e4a268aa38af02480dd0b623f9fa9"}
Jul 30 17:01:35.943341 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:35.943193    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="53c926f82808db4cf291f741c6ddc88114c67e74b2b65dc2e91618215a68645f" exitCode=0
Jul 30 17:01:35.943341 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:35.943265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"53c926f82808db4cf291f741c6ddc88114c67e74b2b65dc2e91618215a68645f"}
Jul 30 17:01:36.948394 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:36.948360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzkvn" event={"ID":"22db70d2-fa87-4b62-99ff-38988b3e2e63","Type":"ContainerStarted","Data":"d7dc9d1cf0deddbc5dc448fe07e144f42c73e0efff843b4075d27459554adcdc"}
Jul 30 17:01:36.948765 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:36.948414    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hzkvn" event={"ID":"22db70d2-fa87-4b62-99ff-38988b3e2e63","Type":"ContainerStarted","Data":"05d806f198265ba09d98f4f93b2c1f50448b839f6500dee2c61f8c6cb2c2e179"}
Jul 30 17:01:36.971564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:36.971516    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hzkvn" podStartSLOduration=3.971479187 podStartE2EDuration="5.971501355s" totalImagesPullingTime="1.92916s" totalInitContainerRuntime="70.862168ms" isStatefulPod=true podCreationTimestamp="2026-07-30 17:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:36.970965168 +0000 UTC m=+65.003775502" watchObservedRunningTime="2026-07-30 17:01:36.971501355 +0000 UTC m=+65.004311692"
Jul 30 17:01:37.654947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.654924    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:37.655048 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.654987    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:37.660531 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.660510    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:37.955185 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.955150    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"984dc94acdcfe41e60e35d487e26e4e9ab0eaa97d678d20a715446e9ecec20c3"}
Jul 30 17:01:37.955579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.955272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"d91eedff36b722428e4121d8d02f4eecc45e20c089363cd4169ede34bfe09751"}
Jul 30 17:01:37.955579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.955306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"0d186d7957be7915168ed5a5bee4e63bd49e0c0d3a0c709ce053de2706963cef"}
Jul 30 17:01:37.955579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.955333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"546e0b184cf97a5c265f5cca93b206d7cf848ef670d959788fb26184e7fc5419"}
Jul 30 17:01:37.955579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.955365    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"f1035da774e36ef8b25c74318caef77bfb708109adfb254b5054115eb07f1c35"}
Jul 30 17:01:37.959092 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.959070    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:01:37.963580 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.963558    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:01:37.966277 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.966249    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:37.969338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969319    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 30 17:01:37.969425 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969321    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 30 17:01:37.969500 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969321    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 30 17:01:37.969760 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969744    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 30 17:01:37.969760 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969761    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 30 17:01:37.969903 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969824    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 30 17:01:37.969903 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.969834    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 30 17:01:37.970050 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970034    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 30 17:01:37.970150 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970131    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 30 17:01:37.970215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970183    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 30 17:01:37.970215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970139    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 30 17:01:37.970450 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970429    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 30 17:01:37.970610 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970594    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tskxn\""
Jul 30 17:01:37.970688 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970672    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7lo0safpda175\""
Jul 30 17:01:37.970780 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970758    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 30 17:01:37.970850 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.970836    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 30 17:01:37.971958 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.971929    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 30 17:01:37.986002 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:37.985977    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:01:38.017355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017505 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017464    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017557 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017557 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017649 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017573    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017854 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.017942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018044 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.017971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018044 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018044 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018066    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018178    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018205    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018237    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mb4n\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-kube-api-access-5mb4n\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-config\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.018455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.018425    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119480 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119448    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119689 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119493    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mb4n\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-kube-api-access-5mb4n\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119689 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-config\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119689 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119785    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.119847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119949    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.119981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120071    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120443 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120101    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120443 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.120891 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.120767    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.121376 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.121351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.121473 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.121432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.121473 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.121465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.121591 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.121501    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.122673 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.122630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.122883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.122854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.122957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.122945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.123020 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.122972    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.123546 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.123521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.123917 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.123891    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.124227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.124202    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.124328 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.124205    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.124735 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.124710    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-config\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.124812 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.124754    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.124878 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.124858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.124927 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.124894    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.125049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.125030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.125109 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.125068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.126595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.126576    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.127391 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.127372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mb4n\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-kube-api-access-5mb4n\") pod \"prometheus-k8s-0\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.222430 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.222398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:38.224876 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.224858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 17:01:38.234944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.234919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5962bc8b-6237-4100-830f-5a54e63e0f14-original-pull-secret\") pod \"global-pull-secret-syncer-2w6gg\" (UID: \"5962bc8b-6237-4100-830f-5a54e63e0f14\") " pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:38.275738 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.275713    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:38.323248 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.323211    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:38.323378 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.323339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:38.325911 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.325885    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 17:01:38.325911 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.325909    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 17:01:38.336573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.336346    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 17:01:38.336573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.336547    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa93763-d018-4c80-aa34-83bc592bc8b8-metrics-certs\") pod \"network-metrics-daemon-qfh2d\" (UID: \"ffa93763-d018-4c80-aa34-83bc592bc8b8\") " pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:38.346757 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.346735    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4sq\" (UniqueName: \"kubernetes.io/projected/9587b1cf-a314-4e21-b752-1cadd4877321-kube-api-access-ml4sq\") pod \"network-check-target-5gzq2\" (UID: \"9587b1cf-a314-4e21-b752-1cadd4877321\") " pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:38.408758 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.408724    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:01:38.411405 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:38.411379    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7f42d1_03d4_4a03_9806_562222f897a7.slice/crio-e99340131b2b9f0550f906c90fe2c448a1a65691e37ec4b3e8db14582f537688 WatchSource:0}: Error finding container e99340131b2b9f0550f906c90fe2c448a1a65691e37ec4b3e8db14582f537688: Status 404 returned error can't find the container with id e99340131b2b9f0550f906c90fe2c448a1a65691e37ec4b3e8db14582f537688
Jul 30 17:01:38.459285 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.459246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2w6gg"
Jul 30 17:01:38.467981 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.467957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wcszl\""
Jul 30 17:01:38.472081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.472063    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4wpnm\""
Jul 30 17:01:38.476933 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.476913    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:38.480294 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.480273    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qfh2d"
Jul 30 17:01:38.627839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.627738    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2w6gg"]
Jul 30 17:01:38.631487 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:38.631452    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5962bc8b_6237_4100_830f_5a54e63e0f14.slice/crio-fa1654f71cdd648c675eec5e25aab060f15859f0e071c13bb749a44c2b7f59f7 WatchSource:0}: Error finding container fa1654f71cdd648c675eec5e25aab060f15859f0e071c13bb749a44c2b7f59f7: Status 404 returned error can't find the container with id fa1654f71cdd648c675eec5e25aab060f15859f0e071c13bb749a44c2b7f59f7
Jul 30 17:01:38.656422 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.656191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5gzq2"]
Jul 30 17:01:38.659060 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:38.659025    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9587b1cf_a314_4e21_b752_1cadd4877321.slice/crio-aca0934086939071ad023539932df5ebf559bb3d4584f2075a54056d1a7b5ddd WatchSource:0}: Error finding container aca0934086939071ad023539932df5ebf559bb3d4584f2075a54056d1a7b5ddd: Status 404 returned error can't find the container with id aca0934086939071ad023539932df5ebf559bb3d4584f2075a54056d1a7b5ddd
Jul 30 17:01:38.681427 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.681397    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qfh2d"]
Jul 30 17:01:38.684937 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:01:38.684823    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa93763_d018_4c80_aa34_83bc592bc8b8.slice/crio-47524d3a40419c22cda80783df5736d81630e07d27d56f874bb51b95f5bc58b8 WatchSource:0}: Error finding container 47524d3a40419c22cda80783df5736d81630e07d27d56f874bb51b95f5bc58b8: Status 404 returned error can't find the container with id 47524d3a40419c22cda80783df5736d81630e07d27d56f874bb51b95f5bc58b8
Jul 30 17:01:38.959564 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.959525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qfh2d" event={"ID":"ffa93763-d018-4c80-aa34-83bc592bc8b8","Type":"ContainerStarted","Data":"47524d3a40419c22cda80783df5736d81630e07d27d56f874bb51b95f5bc58b8"}
Jul 30 17:01:38.960491 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.960468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gzq2" event={"ID":"9587b1cf-a314-4e21-b752-1cadd4877321","Type":"ContainerStarted","Data":"aca0934086939071ad023539932df5ebf559bb3d4584f2075a54056d1a7b5ddd"}
Jul 30 17:01:38.961836 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.961811    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c" exitCode=0
Jul 30 17:01:38.961931 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.961899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"}
Jul 30 17:01:38.961985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.961971    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"e99340131b2b9f0550f906c90fe2c448a1a65691e37ec4b3e8db14582f537688"}
Jul 30 17:01:38.962971 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:38.962947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2w6gg" event={"ID":"5962bc8b-6237-4100-830f-5a54e63e0f14","Type":"ContainerStarted","Data":"fa1654f71cdd648c675eec5e25aab060f15859f0e071c13bb749a44c2b7f59f7"}
Jul 30 17:01:39.973824 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:39.973671    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerStarted","Data":"535e81ead80834a6a7e462177ac156ed4d16ed66cb03216c5ccdfa30b8e89974"}
Jul 30 17:01:40.003423 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:40.002946    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.037052153 podStartE2EDuration="8.002928349s" totalImagesPullingTime="4.951415557s" totalInitContainerRuntime="14.460639ms" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:40.000391188 +0000 UTC m=+68.033201580" watchObservedRunningTime="2026-07-30 17:01:40.002928349 +0000 UTC m=+68.035738706"
Jul 30 17:01:40.982329 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:40.981754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qfh2d" event={"ID":"ffa93763-d018-4c80-aa34-83bc592bc8b8","Type":"ContainerStarted","Data":"2f4864f7a0c416b25f3caf919b7cb35f8714d05aaaebf7aa6eca9dd587183c99"}
Jul 30 17:01:40.982329 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:40.981830    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qfh2d" event={"ID":"ffa93763-d018-4c80-aa34-83bc592bc8b8","Type":"ContainerStarted","Data":"147cadf2aab54ec9f9d960c4bd918e986087dc90589325834fe3c21827894b25"}
Jul 30 17:01:41.005004 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:41.004953    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qfh2d" podStartSLOduration=66.168302543 podStartE2EDuration="1m8.004933101s" totalImagesPullingTime="1.836630558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:41.004140694 +0000 UTC m=+69.036951032" watchObservedRunningTime="2026-07-30 17:01:41.004933101 +0000 UTC m=+69.037743439"
Jul 30 17:01:45.008007 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.007866    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2w6gg" event={"ID":"5962bc8b-6237-4100-830f-5a54e63e0f14","Type":"ContainerStarted","Data":"73c68dcdd3a89fde11875be9caa22d07d77e53460c7fe4838aafa3c356e8abd2"}
Jul 30 17:01:45.011350 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.011312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5gzq2" event={"ID":"9587b1cf-a314-4e21-b752-1cadd4877321","Type":"ContainerStarted","Data":"6687fd783c9a3e4e20bb74c92aeba1b9bf06fde73325c22558f8c51e91338127"}
Jul 30 17:01:45.012303 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.012282    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:01:45.017958 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.017940    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"}
Jul 30 17:01:45.018201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.018180    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"}
Jul 30 17:01:45.033981 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.033011    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2w6gg" podStartSLOduration=65.890460409 podStartE2EDuration="1m12.032994249s" totalImagesPullingTime="6.14253384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:45.032512469 +0000 UTC m=+73.065322807" watchObservedRunningTime="2026-07-30 17:01:45.032994249 +0000 UTC m=+73.065804590"
Jul 30 17:01:45.052124 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:45.052042    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5gzq2" podStartSLOduration=65.949303803 podStartE2EDuration="1m12.052022283s" totalImagesPullingTime="6.10271848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:45.051699366 +0000 UTC m=+73.084509700" watchObservedRunningTime="2026-07-30 17:01:45.052022283 +0000 UTC m=+73.084832621"
Jul 30 17:01:47.031920 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:47.031884    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"}
Jul 30 17:01:47.032329 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:47.031970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"}
Jul 30 17:01:48.038432 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:48.038397    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"}
Jul 30 17:01:48.038802 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:48.038478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerStarted","Data":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"}
Jul 30 17:01:48.070040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:48.069958    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.275868054 podStartE2EDuration="11.069937234s" totalImagesPullingTime="7.769458117s" totalInitContainerRuntime="24.611063ms" isStatefulPod=false podCreationTimestamp="2026-07-30 17:01:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:01:48.06793824 +0000 UTC m=+76.100748578" watchObservedRunningTime="2026-07-30 17:01:48.069937234 +0000 UTC m=+76.102747614"
Jul 30 17:01:48.276847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:48.276811    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:01:54.610416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:01:54.610306    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd4dfffbf-bwkd9"]
Jul 30 17:02:17.034615 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:17.034580    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5gzq2"
Jul 30 17:02:19.629759 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.629717    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dd4dfffbf-bwkd9" podUID="358c72ae-b819-4dee-a147-f94e1e6a9f6a" containerName="console" containerID="cri-o://ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f" gracePeriod=15
Jul 30 17:02:19.859582 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.859555    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:02:19.976446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976408    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-service-ca\") pod \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") "
Jul 30 17:02:19.976446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976453    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-oauth-serving-cert\") pod \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") "
Jul 30 17:02:19.976698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976473    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-serving-cert\") pod \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") "
Jul 30 17:02:19.976698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976497    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-config\") pod \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") "
Jul 30 17:02:19.976698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976536    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlqfg\" (UniqueName: \"kubernetes.io/projected/358c72ae-b819-4dee-a147-f94e1e6a9f6a-kube-api-access-rlqfg\") pod \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") "
Jul 30 17:02:19.976698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976559    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-oauth-config\") pod \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\" (UID: \"358c72ae-b819-4dee-a147-f94e1e6a9f6a\") "
Jul 30 17:02:19.976945 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976912    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-config" (OuterVolumeSpecName: "console-config") pod "358c72ae-b819-4dee-a147-f94e1e6a9f6a" (UID: "358c72ae-b819-4dee-a147-f94e1e6a9f6a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:19.976945 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976925    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "358c72ae-b819-4dee-a147-f94e1e6a9f6a" (UID: "358c72ae-b819-4dee-a147-f94e1e6a9f6a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:19.977052 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.976967    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-service-ca" (OuterVolumeSpecName: "service-ca") pod "358c72ae-b819-4dee-a147-f94e1e6a9f6a" (UID: "358c72ae-b819-4dee-a147-f94e1e6a9f6a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:19.978883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.978857    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "358c72ae-b819-4dee-a147-f94e1e6a9f6a" (UID: "358c72ae-b819-4dee-a147-f94e1e6a9f6a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:19.978981 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.978951    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "358c72ae-b819-4dee-a147-f94e1e6a9f6a" (UID: "358c72ae-b819-4dee-a147-f94e1e6a9f6a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:19.979027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:19.979002    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358c72ae-b819-4dee-a147-f94e1e6a9f6a-kube-api-access-rlqfg" (OuterVolumeSpecName: "kube-api-access-rlqfg") pod "358c72ae-b819-4dee-a147-f94e1e6a9f6a" (UID: "358c72ae-b819-4dee-a147-f94e1e6a9f6a"). InnerVolumeSpecName "kube-api-access-rlqfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:20.077199 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.077167    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqfg\" (UniqueName: \"kubernetes.io/projected/358c72ae-b819-4dee-a147-f94e1e6a9f6a-kube-api-access-rlqfg\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:20.077199 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.077196    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-oauth-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:20.077199 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.077206    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-service-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:20.077416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.077215    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-oauth-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:20.077416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.077223    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-serving-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:20.077416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.077231    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/358c72ae-b819-4dee-a147-f94e1e6a9f6a-console-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:20.167587 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.167551    2569 generic.go:356] "Generic (PLEG): container finished" podID="358c72ae-b819-4dee-a147-f94e1e6a9f6a" containerID="ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f" exitCode=0
Jul 30 17:02:20.167768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.167597    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dd4dfffbf-bwkd9"
Jul 30 17:02:20.167768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.167603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4dfffbf-bwkd9" event={"ID":"358c72ae-b819-4dee-a147-f94e1e6a9f6a","Type":"ContainerDied","Data":"ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f"}
Jul 30 17:02:20.167768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.167646    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dd4dfffbf-bwkd9" event={"ID":"358c72ae-b819-4dee-a147-f94e1e6a9f6a","Type":"ContainerDied","Data":"719a84ce33647055236c773d0d0b1372c2d63f10c40d55e638af07124cd49230"}
Jul 30 17:02:20.167768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.167667    2569 scope.go:117] "RemoveContainer" containerID="ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f"
Jul 30 17:02:20.177049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.177034    2569 scope.go:117] "RemoveContainer" containerID="ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f"
Jul 30 17:02:20.177342 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:20.177316    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f\": container with ID starting with ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f not found: ID does not exist" containerID="ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f"
Jul 30 17:02:20.177435 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.177352    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f"} err="failed to get container status \"ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f\": rpc error: code = NotFound desc = could not find container \"ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f\": container with ID starting with ec0e2e88c2dd7a86515af8d285f5de740f580682c1030b2a65ab18626c66434f not found: ID does not exist"
Jul 30 17:02:20.189544 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.189519    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dd4dfffbf-bwkd9"]
Jul 30 17:02:20.196417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.196393    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dd4dfffbf-bwkd9"]
Jul 30 17:02:20.651962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:20.651929    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358c72ae-b819-4dee-a147-f94e1e6a9f6a" path="/var/lib/kubelet/pods/358c72ae-b819-4dee-a147-f94e1e6a9f6a/volumes"
Jul 30 17:02:38.276461 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:38.276410    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:38.296434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:38.296408    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:39.254288 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:39.254234    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:52.117486 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117450    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:02:52.118067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117739    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="alertmanager" containerID="cri-o://f1035da774e36ef8b25c74318caef77bfb708109adfb254b5054115eb07f1c35" gracePeriod=120
Jul 30 17:02:52.118067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117791    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy" containerID="cri-o://d91eedff36b722428e4121d8d02f4eecc45e20c089363cd4169ede34bfe09751" gracePeriod=120
Jul 30 17:02:52.118067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117825    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="config-reloader" containerID="cri-o://546e0b184cf97a5c265f5cca93b206d7cf848ef670d959788fb26184e7fc5419" gracePeriod=120
Jul 30 17:02:52.118067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117823    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-web" containerID="cri-o://0d186d7957be7915168ed5a5bee4e63bd49e0c0d3a0c709ce053de2706963cef" gracePeriod=120
Jul 30 17:02:52.118067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117865    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-metric" containerID="cri-o://984dc94acdcfe41e60e35d487e26e4e9ab0eaa97d678d20a715446e9ecec20c3" gracePeriod=120
Jul 30 17:02:52.118067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.117768    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="prom-label-proxy" containerID="cri-o://535e81ead80834a6a7e462177ac156ed4d16ed66cb03216c5ccdfa30b8e89974" gracePeriod=120
Jul 30 17:02:52.289912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.289881    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="535e81ead80834a6a7e462177ac156ed4d16ed66cb03216c5ccdfa30b8e89974" exitCode=0
Jul 30 17:02:52.289912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.289905    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="d91eedff36b722428e4121d8d02f4eecc45e20c089363cd4169ede34bfe09751" exitCode=0
Jul 30 17:02:52.289912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.289912    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="546e0b184cf97a5c265f5cca93b206d7cf848ef670d959788fb26184e7fc5419" exitCode=0
Jul 30 17:02:52.289912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.289918    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="f1035da774e36ef8b25c74318caef77bfb708109adfb254b5054115eb07f1c35" exitCode=0
Jul 30 17:02:52.290158 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.289935    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"535e81ead80834a6a7e462177ac156ed4d16ed66cb03216c5ccdfa30b8e89974"}
Jul 30 17:02:52.290158 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.290004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"d91eedff36b722428e4121d8d02f4eecc45e20c089363cd4169ede34bfe09751"}
Jul 30 17:02:52.290158 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.290029    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"546e0b184cf97a5c265f5cca93b206d7cf848ef670d959788fb26184e7fc5419"}
Jul 30 17:02:52.290158 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:52.290055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"f1035da774e36ef8b25c74318caef77bfb708109adfb254b5054115eb07f1c35"}
Jul 30 17:02:53.298212 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.298171    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="984dc94acdcfe41e60e35d487e26e4e9ab0eaa97d678d20a715446e9ecec20c3" exitCode=0
Jul 30 17:02:53.298212 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.298198    2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerID="0d186d7957be7915168ed5a5bee4e63bd49e0c0d3a0c709ce053de2706963cef" exitCode=0
Jul 30 17:02:53.298687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.298227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"984dc94acdcfe41e60e35d487e26e4e9ab0eaa97d678d20a715446e9ecec20c3"}
Jul 30 17:02:53.298687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.298321    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"0d186d7957be7915168ed5a5bee4e63bd49e0c0d3a0c709ce053de2706963cef"}
Jul 30 17:02:53.355792 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.355769    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:53.460040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.459966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-cluster-tls-config\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460010    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460280 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460043    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-metrics-client-ca\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460280 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460070    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-web\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460280 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460198    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460279    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-web-config\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460325    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm62g\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-kube-api-access-hm62g\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460358    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-out\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-volume\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460432    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-tls-assets\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460470    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-main-db\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460496    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-main-tls\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460511    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:53.460714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460524    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy\") pod \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\" (UID: \"8fbe0826-f5fa-402b-9d1f-8f6c13c79709\") "
Jul 30 17:02:53.460714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460597    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:53.460964 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460800    2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-metrics-client-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.460964 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.460823    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.461196 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.461170    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:02:53.463810 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.463763    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-kube-api-access-hm62g" (OuterVolumeSpecName: "kube-api-access-hm62g") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "kube-api-access-hm62g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:53.463925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.463826    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.464201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.464176    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:53.464303 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.464219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.464377 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.464298    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.464578 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.464558    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.464705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.464598    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-out" (OuterVolumeSpecName: "config-out") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:02:53.464803 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.464783    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-volume" (OuterVolumeSpecName: "config-volume") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.467069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.467041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.473398 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.473377    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-web-config" (OuterVolumeSpecName: "web-config") pod "8fbe0826-f5fa-402b-9d1f-8f6c13c79709" (UID: "8fbe0826-f5fa-402b-9d1f-8f6c13c79709"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:53.561963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.561930    2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-volume\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.561963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.561956    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-tls-assets\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.561963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.561967    2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-alertmanager-main-db\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.561976    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-main-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.561986    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.561996    2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-cluster-tls-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.562004    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.562013    2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.562022    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-web-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.562030    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm62g\" (UniqueName: \"kubernetes.io/projected/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-kube-api-access-hm62g\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:53.562155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:53.562038    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fbe0826-f5fa-402b-9d1f-8f6c13c79709-config-out\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:54.305068 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.305026    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fbe0826-f5fa-402b-9d1f-8f6c13c79709","Type":"ContainerDied","Data":"27b0d6e3b7932d1fb27c889fc389936d49c6fc43b33d9f94b35cec2293bcacf4"}
Jul 30 17:02:54.305068 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.305055    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 30 17:02:54.305528 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.305124    2569 scope.go:117] "RemoveContainer" containerID="535e81ead80834a6a7e462177ac156ed4d16ed66cb03216c5ccdfa30b8e89974"
Jul 30 17:02:54.313505 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.313424    2569 scope.go:117] "RemoveContainer" containerID="984dc94acdcfe41e60e35d487e26e4e9ab0eaa97d678d20a715446e9ecec20c3"
Jul 30 17:02:54.320713 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.320690    2569 scope.go:117] "RemoveContainer" containerID="d91eedff36b722428e4121d8d02f4eecc45e20c089363cd4169ede34bfe09751"
Jul 30 17:02:54.327055 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.327037    2569 scope.go:117] "RemoveContainer" containerID="0d186d7957be7915168ed5a5bee4e63bd49e0c0d3a0c709ce053de2706963cef"
Jul 30 17:02:54.332925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.332888    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:02:54.334055 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.334031    2569 scope.go:117] "RemoveContainer" containerID="546e0b184cf97a5c265f5cca93b206d7cf848ef670d959788fb26184e7fc5419"
Jul 30 17:02:54.338168 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.338145    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 30 17:02:54.341502 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.341485    2569 scope.go:117] "RemoveContainer" containerID="f1035da774e36ef8b25c74318caef77bfb708109adfb254b5054115eb07f1c35"
Jul 30 17:02:54.347999 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.347983    2569 scope.go:117] "RemoveContainer" containerID="53c926f82808db4cf291f741c6ddc88114c67e74b2b65dc2e91618215a68645f"
Jul 30 17:02:54.651698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:54.651601    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" path="/var/lib/kubelet/pods/8fbe0826-f5fa-402b-9d1f-8f6c13c79709/volumes"
Jul 30 17:02:56.449667 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.449634    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:02:56.450085 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.449920    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="prometheus" containerID="cri-o://cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3" gracePeriod=600
Jul 30 17:02:56.450085 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.449936    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy" containerID="cri-o://00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7" gracePeriod=600
Jul 30 17:02:56.450085 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.449985    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="config-reloader" containerID="cri-o://c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4" gracePeriod=600
Jul 30 17:02:56.450085 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.449982    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-web" containerID="cri-o://ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57" gracePeriod=600
Jul 30 17:02:56.450085 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.449982    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="thanos-sidecar" containerID="cri-o://7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb" gracePeriod=600
Jul 30 17:02:56.450331 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.450003    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28" gracePeriod=600
Jul 30 17:02:56.682974 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.682947    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:56.782523 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782442    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-1\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782523 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782482    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-config-out\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782523 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782501    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-0\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-web-config\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782546    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-grpc-tls\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-kubelet-serving-ca-bundle\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-kube-rbac-proxy\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782650    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-metrics-client-certs\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782701    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-thanos-prometheus-http-client-file\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782736    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-config\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-trusted-ca-bundle\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.782809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782796    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-tls-assets\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782826    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-serving-certs-ca-bundle\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782857    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782892    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782903    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-tls\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.782975    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-2\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783024    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-metrics-client-ca\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783051    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mb4n\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-kube-api-access-5mb4n\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-db\") pod \"1f7f42d1-03d4-4a03-9806-562222f897a7\" (UID: \"1f7f42d1-03d4-4a03-9806-562222f897a7\") "
Jul 30 17:02:56.783317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783198    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.783750 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783498    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.783750 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783523    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.783873 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.783844    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.784079 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.784047    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.784716 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.784440    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:02:56.784716 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.784594    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.784886 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.784831    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.785220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.785178    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:02:56.785910 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.785621    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.785910 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.785816    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.785910 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.785851    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-config-out" (OuterVolumeSpecName: "config-out") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:02:56.786892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.786866    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.787612 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.787573    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.787612 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.787576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:56.787750 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.787649    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-config" (OuterVolumeSpecName: "config") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.787750 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.787727    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.787894 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.787871    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-kube-api-access-5mb4n" (OuterVolumeSpecName: "kube-api-access-5mb4n") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "kube-api-access-5mb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:02:56.788303 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.788275    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.788418 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.788397    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.795091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.795069    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-web-config" (OuterVolumeSpecName: "web-config") pod "1f7f42d1-03d4-4a03-9806-562222f897a7" (UID: "1f7f42d1-03d4-4a03-9806-562222f897a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:02:56.884001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.883969    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.883994    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884004    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-metrics-client-ca\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884014    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mb4n\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-kube-api-access-5mb4n\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884023    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-db\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884034    2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f7f42d1-03d4-4a03-9806-562222f897a7-config-out\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884042    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884051    2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-web-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884059    2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-grpc-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884067    2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-kube-rbac-proxy\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884075    2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-metrics-client-certs\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884085    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884095    2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884104    2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884113    2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884121    2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f7f42d1-03d4-4a03-9806-562222f897a7-tls-assets\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884129    2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7f42d1-03d4-4a03-9806-562222f897a7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:56.884188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:56.884137    2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1f7f42d1-03d4-4a03-9806-562222f897a7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:02:57.320666 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320626    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28" exitCode=0
Jul 30 17:02:57.320666 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320660    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7" exitCode=0
Jul 30 17:02:57.320666 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320672    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57" exitCode=0
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320681    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb" exitCode=0
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320690    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4" exitCode=0
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320699    2569 generic.go:356] "Generic (PLEG): container finished" podID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3" exitCode=0
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"}
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320725    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"}
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320797    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"}
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320822    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"}
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320837    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320846    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"}
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"}
Jul 30 17:02:57.320969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.320893    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1f7f42d1-03d4-4a03-9806-562222f897a7","Type":"ContainerDied","Data":"e99340131b2b9f0550f906c90fe2c448a1a65691e37ec4b3e8db14582f537688"}
Jul 30 17:02:57.329645 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.329618    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.336160 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.336145    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.342230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.342216    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.348480 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.348458    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:02:57.349010 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.348990    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.353005 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.352985    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:02:57.355692 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.355674    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.362166 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.362151    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.368308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.368290    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.368555 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.368536    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.368633 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.368567    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"} err="failed to get container status \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist"
Jul 30 17:02:57.368633 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.368595    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.368835 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.368818    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.368874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.368841    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"} err="failed to get container status \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist"
Jul 30 17:02:57.368874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.368857    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.369089 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.369072    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.369135 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369094    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"} err="failed to get container status \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist"
Jul 30 17:02:57.369135 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369108    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.369306 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.369290    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.369355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369308    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"} err="failed to get container status \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist"
Jul 30 17:02:57.369355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369321    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.369521 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.369494    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.369601 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369530    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"} err="failed to get container status \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist"
Jul 30 17:02:57.369601 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369562    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.369805 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.369788    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.369858 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369808    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"} err="failed to get container status \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist"
Jul 30 17:02:57.369858 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.369824    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.370146 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:02:57.370073    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.370146 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370123    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"} err="failed to get container status \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist"
Jul 30 17:02:57.370146 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370144    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.370419 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370397    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"} err="failed to get container status \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist"
Jul 30 17:02:57.370495 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370423    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.370681 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370663    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"} err="failed to get container status \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist"
Jul 30 17:02:57.370740 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370682    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.370919 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370900    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"} err="failed to get container status \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist"
Jul 30 17:02:57.370919 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.370919    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.371136 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371119    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"} err="failed to get container status \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist"
Jul 30 17:02:57.371180 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371136    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.371434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371416    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"} err="failed to get container status \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist"
Jul 30 17:02:57.371434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371433    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.371625 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371609    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"} err="failed to get container status \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist"
Jul 30 17:02:57.371666 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371626    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.371827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371810    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"} err="failed to get container status \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist"
Jul 30 17:02:57.371827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371826    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.371995 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371980    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"} err="failed to get container status \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist"
Jul 30 17:02:57.372032 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.371995    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.372203 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372186    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"} err="failed to get container status \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist"
Jul 30 17:02:57.372242 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372203    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.372402 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372386    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"} err="failed to get container status \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist"
Jul 30 17:02:57.372449 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372402    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.372582 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372563    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"} err="failed to get container status \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist"
Jul 30 17:02:57.372629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372588    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.372801 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372784    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"} err="failed to get container status \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist"
Jul 30 17:02:57.372861 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372801    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.373011 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.372995    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"} err="failed to get container status \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist"
Jul 30 17:02:57.373057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373012    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.373189 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373173    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"} err="failed to get container status \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist"
Jul 30 17:02:57.373229 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373188    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.373380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373362    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"} err="failed to get container status \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist"
Jul 30 17:02:57.373428 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373382    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.373547 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373528    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"} err="failed to get container status \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist"
Jul 30 17:02:57.373599 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373549    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.373746 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373724    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"} err="failed to get container status \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist"
Jul 30 17:02:57.373785 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373746    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.373924 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373907    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"} err="failed to get container status \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist"
Jul 30 17:02:57.373970 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.373925    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.374152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374135    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"} err="failed to get container status \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist"
Jul 30 17:02:57.374192 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374153    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.374351 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374333    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"} err="failed to get container status \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist"
Jul 30 17:02:57.374401 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374351    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.374583 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374565    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"} err="failed to get container status \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist"
Jul 30 17:02:57.374631 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374583    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.374779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374763    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"} err="failed to get container status \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist"
Jul 30 17:02:57.374819 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374779    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.374992 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374978    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"} err="failed to get container status \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist"
Jul 30 17:02:57.375029 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.374992    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.375174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375158    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"} err="failed to get container status \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist"
Jul 30 17:02:57.375213 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375174    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.375435 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375419    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"} err="failed to get container status \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist"
Jul 30 17:02:57.375435 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375435    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.375607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375593    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"} err="failed to get container status \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist"
Jul 30 17:02:57.375665 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375607    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.375814 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375798    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"} err="failed to get container status \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist"
Jul 30 17:02:57.375853 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.375814    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.376023 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376006    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"} err="failed to get container status \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist"
Jul 30 17:02:57.376063 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376026    2569 scope.go:117] "RemoveContainer" containerID="8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"
Jul 30 17:02:57.376211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376195    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28"} err="failed to get container status \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": rpc error: code = NotFound desc = could not find container \"8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28\": container with ID starting with 8b0685df35ef055329937a1254d1656256e0c381aaccb282080a0c1155a0be28 not found: ID does not exist"
Jul 30 17:02:57.376211 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376209    2569 scope.go:117] "RemoveContainer" containerID="00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"
Jul 30 17:02:57.376428 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376406    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7"} err="failed to get container status \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": rpc error: code = NotFound desc = could not find container \"00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7\": container with ID starting with 00910cf96301ba10e5432a4edb53cb255a10a7c9a45739a560030d65a17d54b7 not found: ID does not exist"
Jul 30 17:02:57.376428 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376419    2569 scope.go:117] "RemoveContainer" containerID="ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"
Jul 30 17:02:57.376615 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376589    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57"} err="failed to get container status \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": rpc error: code = NotFound desc = could not find container \"ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57\": container with ID starting with ae5b2af8cf7da06e9812bf4a9d9ca558a1abe3b623ba6c10ad645054478e2c57 not found: ID does not exist"
Jul 30 17:02:57.376615 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376605    2569 scope.go:117] "RemoveContainer" containerID="7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"
Jul 30 17:02:57.376802 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376788    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb"} err="failed to get container status \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": rpc error: code = NotFound desc = could not find container \"7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb\": container with ID starting with 7525675440b05158e0255f8706e679b9b625697b14c540ed5171940cc583c1eb not found: ID does not exist"
Jul 30 17:02:57.376802 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376801    2569 scope.go:117] "RemoveContainer" containerID="c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"
Jul 30 17:02:57.377006 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.376992    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4"} err="failed to get container status \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": rpc error: code = NotFound desc = could not find container \"c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4\": container with ID starting with c2056ee12ec3f3be364b5a7acde55ad7f4aec82ec4a72ffb478a350ec19b45d4 not found: ID does not exist"
Jul 30 17:02:57.377006 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.377005    2569 scope.go:117] "RemoveContainer" containerID="cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"
Jul 30 17:02:57.377188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.377174    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3"} err="failed to get container status \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": rpc error: code = NotFound desc = could not find container \"cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3\": container with ID starting with cdd4e4174d0b4a682dcb24337df4b2b5358279432e969ec165d7c36b8bd65dd3 not found: ID does not exist"
Jul 30 17:02:57.377232 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.377189    2569 scope.go:117] "RemoveContainer" containerID="899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"
Jul 30 17:02:57.377386 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.377372    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c"} err="failed to get container status \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": rpc error: code = NotFound desc = could not find container \"899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c\": container with ID starting with 899b541842bde9ca97365a9d380b8135441c558e714159bfe2ca03718a33cc9c not found: ID does not exist"
Jul 30 17:02:57.380826 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.380807    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:02:57.381025 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381014    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="config-reloader"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381027    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="config-reloader"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381036    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381041    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381050    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="prometheus"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381055    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="prometheus"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381061    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="thanos-sidecar"
Jul 30 17:02:57.381064 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381067    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="thanos-sidecar"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381075    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="init-config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381081    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="init-config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381089    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="init-config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381094    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="init-config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381099    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-metric"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381104    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-metric"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381114    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="alertmanager"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381119    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="alertmanager"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381125    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-web"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381131    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-web"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381137    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-thanos"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381142    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-thanos"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381149    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-web"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381154    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-web"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381161    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381165    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381171    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358c72ae-b819-4dee-a147-f94e1e6a9f6a" containerName="console"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381175    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="358c72ae-b819-4dee-a147-f94e1e6a9f6a" containerName="console"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381181    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381186    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381193    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="prom-label-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381197    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="prom-label-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381232    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381239    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381244    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="alertmanager"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381249    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-web"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381271    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="config-reloader"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381289    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="thanos-sidecar"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381296    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381301    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="kube-rbac-proxy-metric"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381305    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fbe0826-f5fa-402b-9d1f-8f6c13c79709" containerName="prom-label-proxy"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381311    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="358c72ae-b819-4dee-a147-f94e1e6a9f6a" containerName="console"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381317    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-web"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381323    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="prometheus"
Jul 30 17:02:57.381308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.381328    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" containerName="kube-rbac-proxy-thanos"
Jul 30 17:02:57.384841 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.384826    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.387777 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.387759    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 30 17:02:57.387906 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.387801    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 30 17:02:57.387906 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.387849    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 30 17:02:57.388039 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388024    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7lo0safpda175\""
Jul 30 17:02:57.388107 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388041    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 30 17:02:57.388107 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388058    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tskxn\""
Jul 30 17:02:57.388186 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388124    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 30 17:02:57.388186 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388125    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 30 17:02:57.388285 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388199    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 30 17:02:57.388406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388390    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 30 17:02:57.388519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388505    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 30 17:02:57.388647 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388621    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 30 17:02:57.388647 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388630    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 30 17:02:57.388848 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.388831    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 30 17:02:57.389169 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.389150    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 30 17:02:57.391366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.391339    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 30 17:02:57.394531 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.394510    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 30 17:02:57.400496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.400477    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:02:57.487776 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487745    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b30290a-45b2-459f-ab8d-2ff3647a442c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.487776 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbq66\" (UniqueName: \"kubernetes.io/projected/0b30290a-45b2-459f-ab8d-2ff3647a442c-kube-api-access-jbq66\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487802    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487849    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b30290a-45b2-459f-ab8d-2ff3647a442c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.487958    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488011    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488089    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488107    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488173 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488122    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488200    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-config\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488304    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488320    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.488552 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.488337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589535 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589451    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589535 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589704 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589704 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589704 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589704 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.589919 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590092 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.589865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590790 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590317    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590790 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590411    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590790 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-config\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590790 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590790 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590514    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.590790 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.591163 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.590803    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592237    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592397    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b30290a-45b2-459f-ab8d-2ff3647a442c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592538    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbq66\" (UniqueName: \"kubernetes.io/projected/0b30290a-45b2-459f-ab8d-2ff3647a442c-kube-api-access-jbq66\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.592610    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b30290a-45b2-459f-ab8d-2ff3647a442c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.593114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.593800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.593666    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.594959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.595098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.595390    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.595649    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.595684    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b30290a-45b2-459f-ab8d-2ff3647a442c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.595783    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.596883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.596117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.597794 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.597480    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.598518 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.598497    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.598636 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.598609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.600154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.598800    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b30290a-45b2-459f-ab8d-2ff3647a442c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.600154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.599142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.600693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.600596    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b30290a-45b2-459f-ab8d-2ff3647a442c-config\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.601177 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.601159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b30290a-45b2-459f-ab8d-2ff3647a442c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.603381 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.603230    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbq66\" (UniqueName: \"kubernetes.io/projected/0b30290a-45b2-459f-ab8d-2ff3647a442c-kube-api-access-jbq66\") pod \"prometheus-k8s-0\" (UID: \"0b30290a-45b2-459f-ab8d-2ff3647a442c\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.695393 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.695342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:02:57.832787 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:57.832618    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 30 17:02:57.835410 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:02:57.835383    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b30290a_45b2_459f_ab8d_2ff3647a442c.slice/crio-e7eeaad54e06af478ee3fb323ae3e11a65a1dcffbb12fb8faba2120642512431 WatchSource:0}: Error finding container e7eeaad54e06af478ee3fb323ae3e11a65a1dcffbb12fb8faba2120642512431: Status 404 returned error can't find the container with id e7eeaad54e06af478ee3fb323ae3e11a65a1dcffbb12fb8faba2120642512431
Jul 30 17:02:58.326828 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:58.326797    2569 generic.go:356] "Generic (PLEG): container finished" podID="0b30290a-45b2-459f-ab8d-2ff3647a442c" containerID="30532675af8fb37ef9d5af25cc0fc70ed3b88e18e046d8162b78c6bf05143bc1" exitCode=0
Jul 30 17:02:58.327000 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:58.326853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerDied","Data":"30532675af8fb37ef9d5af25cc0fc70ed3b88e18e046d8162b78c6bf05143bc1"}
Jul 30 17:02:58.327000 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:58.326903    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"e7eeaad54e06af478ee3fb323ae3e11a65a1dcffbb12fb8faba2120642512431"}
Jul 30 17:02:58.653229 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:58.653200    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7f42d1-03d4-4a03-9806-562222f897a7" path="/var/lib/kubelet/pods/1f7f42d1-03d4-4a03-9806-562222f897a7/volumes"
Jul 30 17:02:59.334123 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.334088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"14ab695ddeec57611925617384b23323276669a27a4d678ed85edd553f433867"}
Jul 30 17:02:59.334290 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.334151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"6f44251797e7ca3bc1c0a0b0af29ae5ddb1a2660a12ff4b5c85f8c789c539240"}
Jul 30 17:02:59.334290 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.334219    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"a1666496dc70569ed62fca253c2c406457bcc76ac8dab5e40ca0cbad5acd8258"}
Jul 30 17:02:59.334290 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.334244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"f76b40c171de24fee1418e1f4e666be02175de4840903e834ea3100320787521"}
Jul 30 17:02:59.334407 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.334293    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"5c763f96e8b1135c9a3949a7d2c0953ff93f323de1f3400ef02535d258885d9f"}
Jul 30 17:02:59.334407 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.334323    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b30290a-45b2-459f-ab8d-2ff3647a442c","Type":"ContainerStarted","Data":"b00ba3e1d93b6cf79a242664fd3daaf6924cd8ed853ce9e0ff9a8fe81446d733"}
Jul 30 17:02:59.367080 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:02:59.366991    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.344828497 podStartE2EDuration="2.366972555s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.144058ms" isStatefulPod=false podCreationTimestamp="2026-07-30 17:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:02:59.364877543 +0000 UTC m=+147.397687879" watchObservedRunningTime="2026-07-30 17:02:59.366972555 +0000 UTC m=+147.399782894"
Jul 30 17:03:02.696369 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:03:02.696332    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:03:57.695872 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:03:57.695824    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:03:57.710900 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:03:57.710875    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:03:58.555446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:03:58.555414    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 30 17:05:32.404289 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:05:32.404246    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 17:06:55.267895 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.267845    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-6cpm8"]
Jul 30 17:06:55.270931 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.270914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.273577 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.273547    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-2ln8r\""
Jul 30 17:06:55.273577 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.273571    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 30 17:06:55.274458 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.274440    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 30 17:06:55.274458 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.274451    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 30 17:06:55.304197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.304169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-6cpm8"]
Jul 30 17:06:55.345913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.345871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-cert\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.346078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.345926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszq9\" (UniqueName: \"kubernetes.io/projected/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-kube-api-access-nszq9\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.446709 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.446671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszq9\" (UniqueName: \"kubernetes.io/projected/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-kube-api-access-nszq9\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.446884 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.446749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-cert\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.446884 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:06:55.446836    2569 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 30 17:06:55.446953 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:06:55.446886    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-cert podName:8484bfc2-ecb1-4344-9a98-a59b27ecd7e7 nodeName:}" failed. No retries permitted until 2026-07-30 17:06:55.946871152 +0000 UTC m=+383.979681467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-cert") pod "llmisvc-controller-manager-654c557856-6cpm8" (UID: "8484bfc2-ecb1-4344-9a98-a59b27ecd7e7") : secret "llmisvc-webhook-server-cert" not found
Jul 30 17:06:55.460398 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.460368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszq9\" (UniqueName: \"kubernetes.io/projected/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-kube-api-access-nszq9\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.950155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.950120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-cert\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:55.952482 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:55.952457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8484bfc2-ecb1-4344-9a98-a59b27ecd7e7-cert\") pod \"llmisvc-controller-manager-654c557856-6cpm8\" (UID: \"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7\") " pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:56.180074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:56.180039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:56.312742 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:56.312702    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-6cpm8"]
Jul 30 17:06:56.317074 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:06:56.317041    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8484bfc2_ecb1_4344_9a98_a59b27ecd7e7.slice/crio-359e8266aae0cdca3f06cc3f94529014fce3e84b8d029e76f24c32d54d224347 WatchSource:0}: Error finding container 359e8266aae0cdca3f06cc3f94529014fce3e84b8d029e76f24c32d54d224347: Status 404 returned error can't find the container with id 359e8266aae0cdca3f06cc3f94529014fce3e84b8d029e76f24c32d54d224347
Jul 30 17:06:56.318461 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:56.318445    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:06:57.135833 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:57.135795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8" event={"ID":"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7","Type":"ContainerStarted","Data":"359e8266aae0cdca3f06cc3f94529014fce3e84b8d029e76f24c32d54d224347"}
Jul 30 17:06:59.146677 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:59.146587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8" event={"ID":"8484bfc2-ecb1-4344-9a98-a59b27ecd7e7","Type":"ContainerStarted","Data":"c61b444bbbdfe291405cac701d57267019732646806beceb31726a54103cb76b"}
Jul 30 17:06:59.147049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:59.146694    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:06:59.164778 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:06:59.164723    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8" podStartSLOduration=1.6417552039999999 podStartE2EDuration="4.164708126s" totalImagesPullingTime="2.522952922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:06:59.163403785 +0000 UTC m=+387.196220976" watchObservedRunningTime="2026-07-30 17:06:59.164708126 +0000 UTC m=+387.197518462"
Jul 30 17:07:30.152538 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:07:30.152507    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-6cpm8"
Jul 30 17:08:27.045853 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.045815    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cnszb"]
Jul 30 17:08:27.049018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.049002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.051537 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.051516    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 30 17:08:27.051652 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.051517    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cmd5r\""
Jul 30 17:08:27.056651 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.056632    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cnszb"]
Jul 30 17:08:27.119019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.118977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-model-cache\") pod \"s3-init-cnszb\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") " pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.119182 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.119086    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxlb\" (UniqueName: \"kubernetes.io/projected/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-kube-api-access-6bxlb\") pod \"s3-init-cnszb\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") " pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.220118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.220082    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxlb\" (UniqueName: \"kubernetes.io/projected/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-kube-api-access-6bxlb\") pod \"s3-init-cnszb\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") " pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.220225 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.220138    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-model-cache\") pod \"s3-init-cnszb\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") " pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.220572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.220554    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-model-cache\") pod \"s3-init-cnszb\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") " pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.229906 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.229889    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxlb\" (UniqueName: \"kubernetes.io/projected/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-kube-api-access-6bxlb\") pod \"s3-init-cnszb\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") " pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.365385 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.365287    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cnszb"
Jul 30 17:08:27.486362 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:27.486330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cnszb"]
Jul 30 17:08:27.489435 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:08:27.489405    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod813e4df1_d0c3_4a69_ae4e_e756cd6268fe.slice/crio-958e9cae6da1fe35f1e548ecd469aa6a52f5aa9eede2088f95a51c823b306ecb WatchSource:0}: Error finding container 958e9cae6da1fe35f1e548ecd469aa6a52f5aa9eede2088f95a51c823b306ecb: Status 404 returned error can't find the container with id 958e9cae6da1fe35f1e548ecd469aa6a52f5aa9eede2088f95a51c823b306ecb
Jul 30 17:08:28.454144 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:28.454106    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cnszb" event={"ID":"813e4df1-d0c3-4a69-ae4e-e756cd6268fe","Type":"ContainerStarted","Data":"958e9cae6da1fe35f1e548ecd469aa6a52f5aa9eede2088f95a51c823b306ecb"}
Jul 30 17:08:31.466328 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:31.466293    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cnszb" event={"ID":"813e4df1-d0c3-4a69-ae4e-e756cd6268fe","Type":"ContainerStarted","Data":"b8dcdd0a32ce52d86bb2e22401c6cbbcb8c62f98d13c5094e18ae7abd39c078a"}
Jul 30 17:08:34.478748 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:34.478712    2569 generic.go:356] "Generic (PLEG): container finished" podID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerID="b8dcdd0a32ce52d86bb2e22401c6cbbcb8c62f98d13c5094e18ae7abd39c078a" exitCode=0
Jul 30 17:08:34.479118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:34.478757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cnszb" event={"ID":"813e4df1-d0c3-4a69-ae4e-e756cd6268fe","Type":"ContainerDied","Data":"b8dcdd0a32ce52d86bb2e22401c6cbbcb8c62f98d13c5094e18ae7abd39c078a"}
Jul 30 17:08:38.828053 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:38.828026    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 30 17:08:39.502972 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:39.502931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cnszb" event={"ID":"813e4df1-d0c3-4a69-ae4e-e756cd6268fe","Type":"ContainerStarted","Data":"551622e6a92c17b4230c7e040e814a20f2cd457a11b9d19a593e21a96f83c055"}
Jul 30 17:08:39.521264 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:08:39.521197    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cnszb" podStartSLOduration=1.573832752 podStartE2EDuration="12.521176986s" totalImagesPullingTime="7.661194991s" totalInitContainerRuntime="3.286149243s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:08:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:08:39.519120303 +0000 UTC m=+487.551930641" watchObservedRunningTime="2026-07-30 17:08:39.521176986 +0000 UTC m=+487.553987325"
Jul 30 17:09:43.724281 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:43.724231    2569 generic.go:356] "Generic (PLEG): container finished" podID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerID="551622e6a92c17b4230c7e040e814a20f2cd457a11b9d19a593e21a96f83c055" exitCode=0
Jul 30 17:09:43.724678 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:43.724287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cnszb" event={"ID":"813e4df1-d0c3-4a69-ae4e-e756cd6268fe","Type":"ContainerDied","Data":"551622e6a92c17b4230c7e040e814a20f2cd457a11b9d19a593e21a96f83c055"}
Jul 30 17:09:44.848955 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:44.848932    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cnszb"
Jul 30 17:09:44.921274 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:44.921224    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bxlb\" (UniqueName: \"kubernetes.io/projected/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-kube-api-access-6bxlb\") pod \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") "
Jul 30 17:09:44.921441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:44.921285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-model-cache\") pod \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\" (UID: \"813e4df1-d0c3-4a69-ae4e-e756cd6268fe\") "
Jul 30 17:09:44.923496 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:44.923468    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-kube-api-access-6bxlb" (OuterVolumeSpecName: "kube-api-access-6bxlb") pod "813e4df1-d0c3-4a69-ae4e-e756cd6268fe" (UID: "813e4df1-d0c3-4a69-ae4e-e756cd6268fe"). InnerVolumeSpecName "kube-api-access-6bxlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:09:44.940098 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:44.940072    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "813e4df1-d0c3-4a69-ae4e-e756cd6268fe" (UID: "813e4df1-d0c3-4a69-ae4e-e756cd6268fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:09:45.022235 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:45.022166    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxlb\" (UniqueName: \"kubernetes.io/projected/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-kube-api-access-6bxlb\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:09:45.022235 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:45.022191    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813e4df1-d0c3-4a69-ae4e-e756cd6268fe-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:09:45.732164 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:45.732121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cnszb" event={"ID":"813e4df1-d0c3-4a69-ae4e-e756cd6268fe","Type":"ContainerDied","Data":"958e9cae6da1fe35f1e548ecd469aa6a52f5aa9eede2088f95a51c823b306ecb"}
Jul 30 17:09:45.732164 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:45.732162    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="958e9cae6da1fe35f1e548ecd469aa6a52f5aa9eede2088f95a51c823b306ecb"
Jul 30 17:09:45.732164 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:45.732134    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cnszb"
Jul 30 17:09:52.919456 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.919421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-27ksm"]
Jul 30 17:09:52.919868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.919696    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerName="download-hf-model"
Jul 30 17:09:52.919868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.919707    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerName="download-hf-model"
Jul 30 17:09:52.919868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.919724    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerName="s3-init"
Jul 30 17:09:52.919868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.919730    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerName="s3-init"
Jul 30 17:09:52.919868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.919772    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="813e4df1-d0c3-4a69-ae4e-e756cd6268fe" containerName="s3-init"
Jul 30 17:09:52.921892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.921872    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:52.924159 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.924139    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 30 17:09:52.924241 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.924142    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cmd5r\""
Jul 30 17:09:52.929909 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.929886    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-27ksm"]
Jul 30 17:09:52.981912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.981877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e95a463-493b-4bed-a984-9b2bdb35aa7c-model-cache\") pod \"s3-tls-init-custom-27ksm\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") " pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:52.982067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:52.981922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvg9\" (UniqueName: \"kubernetes.io/projected/0e95a463-493b-4bed-a984-9b2bdb35aa7c-kube-api-access-lfvg9\") pod \"s3-tls-init-custom-27ksm\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") " pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:53.083208 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.083158    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvg9\" (UniqueName: \"kubernetes.io/projected/0e95a463-493b-4bed-a984-9b2bdb35aa7c-kube-api-access-lfvg9\") pod \"s3-tls-init-custom-27ksm\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") " pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:53.083397 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.083271    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e95a463-493b-4bed-a984-9b2bdb35aa7c-model-cache\") pod \"s3-tls-init-custom-27ksm\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") " pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:53.083605 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.083587    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e95a463-493b-4bed-a984-9b2bdb35aa7c-model-cache\") pod \"s3-tls-init-custom-27ksm\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") " pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:53.092804 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.092782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvg9\" (UniqueName: \"kubernetes.io/projected/0e95a463-493b-4bed-a984-9b2bdb35aa7c-kube-api-access-lfvg9\") pod \"s3-tls-init-custom-27ksm\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") " pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:53.239776 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.239752    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:09:53.359869 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.359835    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-27ksm"]
Jul 30 17:09:53.362534 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:09:53.362504    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e95a463_493b_4bed_a984_9b2bdb35aa7c.slice/crio-c00c76f250f3aad1c81eb8b2e01a4f4adf3dbd00e2c39f290e0db45d66d9f706 WatchSource:0}: Error finding container c00c76f250f3aad1c81eb8b2e01a4f4adf3dbd00e2c39f290e0db45d66d9f706: Status 404 returned error can't find the container with id c00c76f250f3aad1c81eb8b2e01a4f4adf3dbd00e2c39f290e0db45d66d9f706
Jul 30 17:09:53.761096 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.761063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-27ksm" event={"ID":"0e95a463-493b-4bed-a984-9b2bdb35aa7c","Type":"ContainerStarted","Data":"89862bc71d1ff35a1e5b4dd82131ccf7969f1126cd2664f4b3f2fcd5524ecee1"}
Jul 30 17:09:53.761096 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:53.761102    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-27ksm" event={"ID":"0e95a463-493b-4bed-a984-9b2bdb35aa7c","Type":"ContainerStarted","Data":"c00c76f250f3aad1c81eb8b2e01a4f4adf3dbd00e2c39f290e0db45d66d9f706"}
Jul 30 17:09:55.769976 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:55.769943    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerID="89862bc71d1ff35a1e5b4dd82131ccf7969f1126cd2664f4b3f2fcd5524ecee1" exitCode=0
Jul 30 17:09:55.770354 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:55.770019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-27ksm" event={"ID":"0e95a463-493b-4bed-a984-9b2bdb35aa7c","Type":"ContainerDied","Data":"89862bc71d1ff35a1e5b4dd82131ccf7969f1126cd2664f4b3f2fcd5524ecee1"}
Jul 30 17:09:56.775163 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:56.775129    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-27ksm" event={"ID":"0e95a463-493b-4bed-a984-9b2bdb35aa7c","Type":"ContainerStarted","Data":"52000a58a62860e4ba79d905554c314ef3d4e9187b1bba5b4d942dd0a43599e5"}
Jul 30 17:09:56.793560 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:09:56.793514    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-27ksm" podStartSLOduration=2.934157319 podStartE2EDuration="4.793499996s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.859342677s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:09:56.792280675 +0000 UTC m=+564.825091013" watchObservedRunningTime="2026-07-30 17:09:56.793499996 +0000 UTC m=+564.826310333"
Jul 30 17:10:01.794755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:01.794727    2569 generic.go:356] "Generic (PLEG): container finished" podID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerID="52000a58a62860e4ba79d905554c314ef3d4e9187b1bba5b4d942dd0a43599e5" exitCode=0
Jul 30 17:10:01.795072 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:01.794786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-27ksm" event={"ID":"0e95a463-493b-4bed-a984-9b2bdb35aa7c","Type":"ContainerDied","Data":"52000a58a62860e4ba79d905554c314ef3d4e9187b1bba5b4d942dd0a43599e5"}
Jul 30 17:10:02.939165 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:02.939139    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:10:03.072443 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.072356    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e95a463-493b-4bed-a984-9b2bdb35aa7c-model-cache\") pod \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") "
Jul 30 17:10:03.072443 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.072415    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfvg9\" (UniqueName: \"kubernetes.io/projected/0e95a463-493b-4bed-a984-9b2bdb35aa7c-kube-api-access-lfvg9\") pod \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\" (UID: \"0e95a463-493b-4bed-a984-9b2bdb35aa7c\") "
Jul 30 17:10:03.074457 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.074424    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e95a463-493b-4bed-a984-9b2bdb35aa7c-kube-api-access-lfvg9" (OuterVolumeSpecName: "kube-api-access-lfvg9") pod "0e95a463-493b-4bed-a984-9b2bdb35aa7c" (UID: "0e95a463-493b-4bed-a984-9b2bdb35aa7c"). InnerVolumeSpecName "kube-api-access-lfvg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:10:03.094545 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.094496    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e95a463-493b-4bed-a984-9b2bdb35aa7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e95a463-493b-4bed-a984-9b2bdb35aa7c" (UID: "0e95a463-493b-4bed-a984-9b2bdb35aa7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:10:03.173923 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.173888    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e95a463-493b-4bed-a984-9b2bdb35aa7c-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:10:03.173923 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.173916    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfvg9\" (UniqueName: \"kubernetes.io/projected/0e95a463-493b-4bed-a984-9b2bdb35aa7c-kube-api-access-lfvg9\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:10:03.803271 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.803238    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-27ksm"
Jul 30 17:10:03.803417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.803232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-27ksm" event={"ID":"0e95a463-493b-4bed-a984-9b2bdb35aa7c","Type":"ContainerDied","Data":"c00c76f250f3aad1c81eb8b2e01a4f4adf3dbd00e2c39f290e0db45d66d9f706"}
Jul 30 17:10:03.803417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:03.803357    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00c76f250f3aad1c81eb8b2e01a4f4adf3dbd00e2c39f290e0db45d66d9f706"
Jul 30 17:10:07.244596 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.244561    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-6hsnc"]
Jul 30 17:10:07.244970 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.244805    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerName="download-hf-model"
Jul 30 17:10:07.244970 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.244816    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerName="download-hf-model"
Jul 30 17:10:07.244970 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.244835    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerName="s3-tls-init-custom"
Jul 30 17:10:07.244970 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.244840    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerName="s3-tls-init-custom"
Jul 30 17:10:07.244970 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.244884    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e95a463-493b-4bed-a984-9b2bdb35aa7c" containerName="s3-tls-init-custom"
Jul 30 17:10:07.247018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.247002    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.249567 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.249543    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cmd5r\""
Jul 30 17:10:07.249703 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.249583    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 30 17:10:07.254720 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.254695    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-6hsnc"]
Jul 30 17:10:07.406620 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.406575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078beaf9-1570-422f-be88-588dfb174d2d-model-cache\") pod \"s3-tls-init-serving-6hsnc\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") " pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.406810 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.406645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8m8c\" (UniqueName: \"kubernetes.io/projected/078beaf9-1570-422f-be88-588dfb174d2d-kube-api-access-v8m8c\") pod \"s3-tls-init-serving-6hsnc\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") " pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.507491 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.507401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078beaf9-1570-422f-be88-588dfb174d2d-model-cache\") pod \"s3-tls-init-serving-6hsnc\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") " pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.507491 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.507463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8m8c\" (UniqueName: \"kubernetes.io/projected/078beaf9-1570-422f-be88-588dfb174d2d-kube-api-access-v8m8c\") pod \"s3-tls-init-serving-6hsnc\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") " pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.507854 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.507832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078beaf9-1570-422f-be88-588dfb174d2d-model-cache\") pod \"s3-tls-init-serving-6hsnc\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") " pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.516456 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.516429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8m8c\" (UniqueName: \"kubernetes.io/projected/078beaf9-1570-422f-be88-588dfb174d2d-kube-api-access-v8m8c\") pod \"s3-tls-init-serving-6hsnc\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") " pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.570444 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.570395    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:07.694433 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.694401    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-6hsnc"]
Jul 30 17:10:07.699015 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:10:07.698977    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod078beaf9_1570_422f_be88_588dfb174d2d.slice/crio-129f892aec96716d7588f9a1734ace98ae8c0c3a8b4dd51e3662ce337684b091 WatchSource:0}: Error finding container 129f892aec96716d7588f9a1734ace98ae8c0c3a8b4dd51e3662ce337684b091: Status 404 returned error can't find the container with id 129f892aec96716d7588f9a1734ace98ae8c0c3a8b4dd51e3662ce337684b091
Jul 30 17:10:07.819218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.819125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6hsnc" event={"ID":"078beaf9-1570-422f-be88-588dfb174d2d","Type":"ContainerStarted","Data":"f1d12bbafadb873c3454dc160c9cf52a74c75d23f43e236cf09f912b8affe7eb"}
Jul 30 17:10:07.819218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:07.819179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6hsnc" event={"ID":"078beaf9-1570-422f-be88-588dfb174d2d","Type":"ContainerStarted","Data":"129f892aec96716d7588f9a1734ace98ae8c0c3a8b4dd51e3662ce337684b091"}
Jul 30 17:10:09.827981 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:09.827945    2569 generic.go:356] "Generic (PLEG): container finished" podID="078beaf9-1570-422f-be88-588dfb174d2d" containerID="f1d12bbafadb873c3454dc160c9cf52a74c75d23f43e236cf09f912b8affe7eb" exitCode=0
Jul 30 17:10:09.828372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:09.828017    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6hsnc" event={"ID":"078beaf9-1570-422f-be88-588dfb174d2d","Type":"ContainerDied","Data":"f1d12bbafadb873c3454dc160c9cf52a74c75d23f43e236cf09f912b8affe7eb"}
Jul 30 17:10:10.834273 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:10.834232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6hsnc" event={"ID":"078beaf9-1570-422f-be88-588dfb174d2d","Type":"ContainerStarted","Data":"52c8a8319a73b56049d19bc9be9c648290d481defc20f19fccd1358f13d1415d"}
Jul 30 17:10:10.852350 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:10.852292    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-6hsnc" podStartSLOduration=2.011759989 podStartE2EDuration="3.852276639s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.84051665s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:10:10.85109107 +0000 UTC m=+578.883901408" watchObservedRunningTime="2026-07-30 17:10:10.852276639 +0000 UTC m=+578.885086970"
Jul 30 17:10:17.861401 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:17.861362    2569 generic.go:356] "Generic (PLEG): container finished" podID="078beaf9-1570-422f-be88-588dfb174d2d" containerID="52c8a8319a73b56049d19bc9be9c648290d481defc20f19fccd1358f13d1415d" exitCode=0
Jul 30 17:10:17.861815 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:17.861437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6hsnc" event={"ID":"078beaf9-1570-422f-be88-588dfb174d2d","Type":"ContainerDied","Data":"52c8a8319a73b56049d19bc9be9c648290d481defc20f19fccd1358f13d1415d"}
Jul 30 17:10:18.999079 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:18.999055    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:19.099860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.099827    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078beaf9-1570-422f-be88-588dfb174d2d-model-cache\") pod \"078beaf9-1570-422f-be88-588dfb174d2d\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") "
Jul 30 17:10:19.100029 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.099881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8m8c\" (UniqueName: \"kubernetes.io/projected/078beaf9-1570-422f-be88-588dfb174d2d-kube-api-access-v8m8c\") pod \"078beaf9-1570-422f-be88-588dfb174d2d\" (UID: \"078beaf9-1570-422f-be88-588dfb174d2d\") "
Jul 30 17:10:19.101903 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.101871    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078beaf9-1570-422f-be88-588dfb174d2d-kube-api-access-v8m8c" (OuterVolumeSpecName: "kube-api-access-v8m8c") pod "078beaf9-1570-422f-be88-588dfb174d2d" (UID: "078beaf9-1570-422f-be88-588dfb174d2d"). InnerVolumeSpecName "kube-api-access-v8m8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:10:19.201327 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.201294    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8m8c\" (UniqueName: \"kubernetes.io/projected/078beaf9-1570-422f-be88-588dfb174d2d-kube-api-access-v8m8c\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:10:19.247516 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.247484    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/078beaf9-1570-422f-be88-588dfb174d2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "078beaf9-1570-422f-be88-588dfb174d2d" (UID: "078beaf9-1570-422f-be88-588dfb174d2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:10:19.302072 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.302042    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/078beaf9-1570-422f-be88-588dfb174d2d-model-cache\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:10:19.870104 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.870060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-6hsnc" event={"ID":"078beaf9-1570-422f-be88-588dfb174d2d","Type":"ContainerDied","Data":"129f892aec96716d7588f9a1734ace98ae8c0c3a8b4dd51e3662ce337684b091"}
Jul 30 17:10:19.870104 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.870099    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="129f892aec96716d7588f9a1734ace98ae8c0c3a8b4dd51e3662ce337684b091"
Jul 30 17:10:19.870104 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:19.870075    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-6hsnc"
Jul 30 17:10:31.563835 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.563799    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"]
Jul 30 17:10:31.564322 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.564204    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078beaf9-1570-422f-be88-588dfb174d2d" containerName="download-hf-model"
Jul 30 17:10:31.564322 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.564222    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="078beaf9-1570-422f-be88-588dfb174d2d" containerName="download-hf-model"
Jul 30 17:10:31.564322 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.564239    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078beaf9-1570-422f-be88-588dfb174d2d" containerName="s3-tls-init-serving"
Jul 30 17:10:31.564322 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.564247    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="078beaf9-1570-422f-be88-588dfb174d2d" containerName="s3-tls-init-serving"
Jul 30 17:10:31.564593 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.564337    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="078beaf9-1570-422f-be88-588dfb174d2d" containerName="s3-tls-init-serving"
Jul 30 17:10:31.566710 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.566690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.569669 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.569641    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 30 17:10:31.569839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.569644    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 30 17:10:31.569839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.569644    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:10:31.569839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.569667    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:10:31.570013 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.569942    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:10:31.570013 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.569949    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:10:31.570115 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.570035    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:10:31.576581 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.576561    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"]
Jul 30 17:10:31.598824 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.598794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.598921 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.598847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.598921 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.598908    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.599004 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.598937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrf2\" (UniqueName: \"kubernetes.io/projected/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-api-access-vkrf2\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.599004 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.598959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.699819 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.699795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.699925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.699849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.699925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.699887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.700023 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.699931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrf2\" (UniqueName: \"kubernetes.io/projected/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-api-access-vkrf2\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.700023 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.699962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.700119 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:10:31.700095    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 30 17:10:31.700231 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:10:31.700194    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls podName:fcaf8b7d-da8b-40ee-8e50-d05f20c520d8 nodeName:}" failed. No retries permitted until 2026-07-30 17:10:32.200171437 +0000 UTC m=+600.232981761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls") pod "isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" (UID: "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 30 17:10:31.700231 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.700220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.700543 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.700526    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.700662 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.700647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:31.708865 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:31.708849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrf2\" (UniqueName: \"kubernetes.io/projected/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-api-access-vkrf2\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:32.203755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.203721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:32.206099 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.206069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:32.480938 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.480865    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:10:32.488517 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.488487    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:32.626978 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.623409    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"]
Jul 30 17:10:32.631005 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:10:32.630979    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcaf8b7d_da8b_40ee_8e50_d05f20c520d8.slice/crio-6944ff0032e717ae27193d4b2fbeaa445f3adfa281947adffe6547fac6e99136 WatchSource:0}: Error finding container 6944ff0032e717ae27193d4b2fbeaa445f3adfa281947adffe6547fac6e99136: Status 404 returned error can't find the container with id 6944ff0032e717ae27193d4b2fbeaa445f3adfa281947adffe6547fac6e99136
Jul 30 17:10:32.635950 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.635924    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:10:32.919028 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.918936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerStarted","Data":"8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b"}
Jul 30 17:10:32.919028 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:32.919011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerStarted","Data":"6944ff0032e717ae27193d4b2fbeaa445f3adfa281947adffe6547fac6e99136"}
Jul 30 17:10:36.934573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:36.934538    2569 generic.go:356] "Generic (PLEG): container finished" podID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerID="8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b" exitCode=0
Jul 30 17:10:36.934944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:36.934601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerDied","Data":"8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b"}
Jul 30 17:10:51.004266 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:51.004227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerStarted","Data":"8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e"}
Jul 30 17:10:53.016052 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:53.016015    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerStarted","Data":"ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f"}
Jul 30 17:10:56.032324 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:56.032286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerStarted","Data":"7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3"}
Jul 30 17:10:56.032698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:56.032484    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:56.032698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:56.032536    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:56.033934 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:56.033909    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:10:57.036317 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:57.036277    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:10:57.036747 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:57.036437    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:57.037402 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:57.037374    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:10:57.040297 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:57.040279    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:10:57.062705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:57.062642    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podStartSLOduration=3.592485566 podStartE2EDuration="26.062629681s" totalImagesPullingTime="18.598192752s" totalInitContainerRuntime="3.871951363s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:10:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:10:56.054927248 +0000 UTC m=+624.087737612" watchObservedRunningTime="2026-07-30 17:10:57.062629681 +0000 UTC m=+625.095440018"
Jul 30 17:10:58.040944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:58.040903    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:10:58.041357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:58.041179    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:10:59.047249 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:59.047200    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:10:59.048367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:10:59.048323    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:09.047740 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:09.047689    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:11:09.048330 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:09.048234    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:19.047754 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:19.047703    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:11:19.048299 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:19.048189    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:29.048267 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:29.048212    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:11:29.048813 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:29.048688    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:39.047607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:39.047553    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:11:39.048098 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:39.047928    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:49.047569 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:49.047522    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:11:49.048046 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:49.048019    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:11:59.047832 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:59.047778    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:11:59.048331 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:11:59.048293    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:08.651760 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:08.651732    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:08.652124 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:08.651806    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:16.713786 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.713740    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"]
Jul 30 17:12:16.714357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.714042    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" containerID="cri-o://8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e" gracePeriod=30
Jul 30 17:12:16.714357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.714101    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" containerID="cri-o://ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f" gracePeriod=30
Jul 30 17:12:16.714357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.714086    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" containerID="cri-o://7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3" gracePeriod=30
Jul 30 17:12:16.733378 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.733353    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"]
Jul 30 17:12:16.736107 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.736090    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.738465 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.738445    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 30 17:12:16.738550 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.738499    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 30 17:12:16.746696 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.746676    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"]
Jul 30 17:12:16.853873 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.853839    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.853873 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.853878    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.854114 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.853952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.854114 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.854014    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqt94\" (UniqueName: \"kubernetes.io/projected/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-api-access-nqt94\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.854114 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.854062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.954654 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.954614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.954654 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.954655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.954860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.954788    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.954860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.954837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqt94\" (UniqueName: \"kubernetes.io/projected/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-api-access-nqt94\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.954945 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.954877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.955028 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.955003    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.955363 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.955346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.955493 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.955474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.957319 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.957303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:16.963653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:16.963634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqt94\" (UniqueName: \"kubernetes.io/projected/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-api-access-nqt94\") pod \"isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:17.036648 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.036551    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 30 17:12:17.046478 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.046184    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:17.211280 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.211236    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"]
Jul 30 17:12:17.213767 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:12:17.213739    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201a827d_5f5c_4cdf_a1a0_48e3cf7b3d3a.slice/crio-685ce9fb8a3422528dc7de3d5b0cedae39308dbe9ec883d0dc34e5d597639f8f WatchSource:0}: Error finding container 685ce9fb8a3422528dc7de3d5b0cedae39308dbe9ec883d0dc34e5d597639f8f: Status 404 returned error can't find the container with id 685ce9fb8a3422528dc7de3d5b0cedae39308dbe9ec883d0dc34e5d597639f8f
Jul 30 17:12:17.339130 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.339031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerStarted","Data":"3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44"}
Jul 30 17:12:17.339325 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.339155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerStarted","Data":"685ce9fb8a3422528dc7de3d5b0cedae39308dbe9ec883d0dc34e5d597639f8f"}
Jul 30 17:12:17.341969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.341945    2569 generic.go:356] "Generic (PLEG): container finished" podID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerID="ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f" exitCode=2
Jul 30 17:12:17.342115 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:17.341990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerDied","Data":"ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f"}
Jul 30 17:12:18.648495 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:18.648449    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:12:18.648898 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:18.648768    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:21.358573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:21.358543    2569 generic.go:356] "Generic (PLEG): container finished" podID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerID="8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e" exitCode=0
Jul 30 17:12:21.358937 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:21.358617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerDied","Data":"8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e"}
Jul 30 17:12:21.359995 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:21.359974    2569 generic.go:356] "Generic (PLEG): container finished" podID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerID="3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44" exitCode=0
Jul 30 17:12:21.360110 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:21.360039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerDied","Data":"3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44"}
Jul 30 17:12:22.036975 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:22.036932    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 30 17:12:22.366749 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:22.366661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerStarted","Data":"0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa"}
Jul 30 17:12:22.366749 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:22.366738    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerStarted","Data":"ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2"}
Jul 30 17:12:22.367215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:22.366786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerStarted","Data":"3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322"}
Jul 30 17:12:22.367215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:22.367106    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:22.391072 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:22.391022    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podStartSLOduration=2.557173787 podStartE2EDuration="6.391007082s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833833295s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:12:22.389661938 +0000 UTC m=+710.422472277" watchObservedRunningTime="2026-07-30 17:12:22.391007082 +0000 UTC m=+710.423817419"
Jul 30 17:12:23.371009 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:23.370974    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:23.371471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:23.371039    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:23.372329 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:23.372301    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:12:23.372921 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:23.372891    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:24.375087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:24.375039    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:12:24.375565 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:24.375503    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:27.036752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:27.036710    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 30 17:12:27.037136 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:27.036833    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:28.647884 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:28.647845    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:12:28.648322 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:28.648279    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:29.379993 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:29.379949    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:12:29.380506 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:29.380469    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:12:29.380819 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:29.380789    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:32.037553 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:32.037501    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 30 17:12:37.037417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:37.037375    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 30 17:12:38.648625 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:38.648575    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 30 17:12:38.649046 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:38.648917    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:38.651817 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:38.651796    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:38.651916 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:38.651883    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:39.380892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:39.380836    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:12:39.381273 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:39.381227    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:42.036556 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:42.036510    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 30 17:12:46.864689 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.864663    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:46.909559 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909532    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-cabundle-cert\") pod \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") "
Jul 30 17:12:46.909725 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909588    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-rbac-proxy-sar-config\") pod \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") "
Jul 30 17:12:46.909725 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909662    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kserve-provision-location\") pod \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") "
Jul 30 17:12:46.909823 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909768    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkrf2\" (UniqueName: \"kubernetes.io/projected/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-api-access-vkrf2\") pod \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") "
Jul 30 17:12:46.909874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909821    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls\") pod \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\" (UID: \"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8\") "
Jul 30 17:12:46.909930 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909901    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" (UID: "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:12:46.909986 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909952    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" (UID: "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:12:46.910042 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.909992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" (UID: "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:12:46.910042 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.910029    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:12:46.910143 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.910046    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:12:46.911977 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.911948    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" (UID: "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:12:46.912080 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:46.911998    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-api-access-vkrf2" (OuterVolumeSpecName: "kube-api-access-vkrf2") pod "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" (UID: "fcaf8b7d-da8b-40ee-8e50-d05f20c520d8"). InnerVolumeSpecName "kube-api-access-vkrf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:12:47.010592 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.010511    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkrf2\" (UniqueName: \"kubernetes.io/projected/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-api-access-vkrf2\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:12:47.010592 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.010543    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:12:47.010592 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.010555    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:12:47.462136 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.462106    2569 generic.go:356] "Generic (PLEG): container finished" podID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerID="7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3" exitCode=0
Jul 30 17:12:47.462323 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.462167    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"
Jul 30 17:12:47.462323 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.462191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerDied","Data":"7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3"}
Jul 30 17:12:47.462323 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.462276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh" event={"ID":"fcaf8b7d-da8b-40ee-8e50-d05f20c520d8","Type":"ContainerDied","Data":"6944ff0032e717ae27193d4b2fbeaa445f3adfa281947adffe6547fac6e99136"}
Jul 30 17:12:47.462323 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.462315    2569 scope.go:117] "RemoveContainer" containerID="7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3"
Jul 30 17:12:47.470091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.470077    2569 scope.go:117] "RemoveContainer" containerID="ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f"
Jul 30 17:12:47.477055 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.477037    2569 scope.go:117] "RemoveContainer" containerID="8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e"
Jul 30 17:12:47.484471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.484451    2569 scope.go:117] "RemoveContainer" containerID="8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b"
Jul 30 17:12:47.485228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.485209    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"]
Jul 30 17:12:47.489736 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.489708    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7d77cd6d86-6jthh"]
Jul 30 17:12:47.492011 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.491995    2569 scope.go:117] "RemoveContainer" containerID="7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3"
Jul 30 17:12:47.492327 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:12:47.492234    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3\": container with ID starting with 7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3 not found: ID does not exist" containerID="7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3"
Jul 30 17:12:47.492327 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.492280    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3"} err="failed to get container status \"7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3\": rpc error: code = NotFound desc = could not find container \"7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3\": container with ID starting with 7bd661ea72301fbc89bf504a09b98a17ecca6e1b477685ee9bd38976ea98a3c3 not found: ID does not exist"
Jul 30 17:12:47.492327 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.492299    2569 scope.go:117] "RemoveContainer" containerID="ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f"
Jul 30 17:12:47.492575 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:12:47.492555    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f\": container with ID starting with ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f not found: ID does not exist" containerID="ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f"
Jul 30 17:12:47.492617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.492582    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f"} err="failed to get container status \"ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f\": rpc error: code = NotFound desc = could not find container \"ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f\": container with ID starting with ccbaeab6322ac0a0a447ee78de561c30070ae64caab44c547ccd70d2283e4b6f not found: ID does not exist"
Jul 30 17:12:47.492617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.492602    2569 scope.go:117] "RemoveContainer" containerID="8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e"
Jul 30 17:12:47.492825 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:12:47.492806    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e\": container with ID starting with 8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e not found: ID does not exist" containerID="8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e"
Jul 30 17:12:47.492867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.492835    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e"} err="failed to get container status \"8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e\": rpc error: code = NotFound desc = could not find container \"8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e\": container with ID starting with 8113c57dd22e2e0d77c82b64af07db8d79df316c8e10c9678ed0a4634eb20c5e not found: ID does not exist"
Jul 30 17:12:47.492867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.492852    2569 scope.go:117] "RemoveContainer" containerID="8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b"
Jul 30 17:12:47.493069 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:12:47.493054    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b\": container with ID starting with 8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b not found: ID does not exist" containerID="8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b"
Jul 30 17:12:47.493125 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:47.493075    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b"} err="failed to get container status \"8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b\": rpc error: code = NotFound desc = could not find container \"8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b\": container with ID starting with 8d229ee9cfe3e008f41b4968fe8ba2c7e3a60fb2058873af503d99cef2b0263b not found: ID does not exist"
Jul 30 17:12:48.652100 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:48.652071    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" path="/var/lib/kubelet/pods/fcaf8b7d-da8b-40ee-8e50-d05f20c520d8/volumes"
Jul 30 17:12:49.380680 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:49.380637    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:12:49.381146 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:49.381123    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:12:59.380786 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:59.380745    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:12:59.381312 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:12:59.381287    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:13:09.381295 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:09.381234    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:13:09.381779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:09.381627    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:13:19.380704 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:19.380656    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:13:19.381207 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:19.381026    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:13:29.380895 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:29.380855    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:13:29.381390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:29.381370    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:13:39.381360 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:39.381326    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:13:39.381746 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:39.381462    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:13:51.920653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:51.920620    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"]
Jul 30 17:13:51.921168 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:51.920889    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" containerID="cri-o://3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322" gracePeriod=30
Jul 30 17:13:51.921168 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:51.920933    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" containerID="cri-o://0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa" gracePeriod=30
Jul 30 17:13:51.921168 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:51.920952    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" containerID="cri-o://ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2" gracePeriod=30
Jul 30 17:13:52.710410 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:52.710378    2569 generic.go:356] "Generic (PLEG): container finished" podID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerID="ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2" exitCode=2
Jul 30 17:13:52.710574 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:52.710437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerDied","Data":"ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2"}
Jul 30 17:13:54.375761 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:54.375674    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 30 17:13:56.729971 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:56.729943    2569 generic.go:356] "Generic (PLEG): container finished" podID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerID="3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322" exitCode=0
Jul 30 17:13:56.730337 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:56.730016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerDied","Data":"3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322"}
Jul 30 17:13:59.375514 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:59.375476    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 30 17:13:59.380839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:59.380813    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:13:59.381229 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:13:59.381203    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:04.375799 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:04.375754    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 30 17:14:04.376275 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:04.375886    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:14:09.375016 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:09.374973    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 30 17:14:09.380449 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:09.380414    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:14:09.380630 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:09.380607    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:11.940678 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940644    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"]
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940933    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940944    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940955    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940961    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940968    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="storage-initializer"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940974    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="storage-initializer"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940988    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.940996    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.941046    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kserve-container"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.941054    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="kube-rbac-proxy"
Jul 30 17:14:11.941118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.941058    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcaf8b7d-da8b-40ee-8e50-d05f20c520d8" containerName="agent"
Jul 30 17:14:11.943347 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.943330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:11.945801 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.945765    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 30 17:14:11.945801 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.945788    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 30 17:14:11.953449 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:11.953426    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"]
Jul 30 17:14:12.090773 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.090735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.090956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.090778    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5pd\" (UniqueName: \"kubernetes.io/projected/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-api-access-mg5pd\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.090956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.090810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af03e7a-542b-4636-9f8d-bf0fc11208bb-proxy-tls\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.090956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.090833    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-cabundle-cert\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.090956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.090880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kserve-provision-location\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.191588 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.191502    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.191588 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.191542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5pd\" (UniqueName: \"kubernetes.io/projected/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-api-access-mg5pd\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.191588 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.191565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af03e7a-542b-4636-9f8d-bf0fc11208bb-proxy-tls\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.191588 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.191582    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-cabundle-cert\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.191901 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.191622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kserve-provision-location\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.192091 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.192067    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kserve-provision-location\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.192307 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.192288    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.192372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.192356    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-cabundle-cert\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.193881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.193863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af03e7a-542b-4636-9f8d-bf0fc11208bb-proxy-tls\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.200485 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.200466    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5pd\" (UniqueName: \"kubernetes.io/projected/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-api-access-mg5pd\") pod \"isvc-logger-predictor-d7459ccb5-lckrg\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.254061 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.254021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:12.382150 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.382086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"]
Jul 30 17:14:12.384354 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:14:12.384327    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af03e7a_542b_4636_9f8d_bf0fc11208bb.slice/crio-a196ee5f4f5702d98104d816ba5520519d4f2ba87462d04b01700f0a6dccc522 WatchSource:0}: Error finding container a196ee5f4f5702d98104d816ba5520519d4f2ba87462d04b01700f0a6dccc522: Status 404 returned error can't find the container with id a196ee5f4f5702d98104d816ba5520519d4f2ba87462d04b01700f0a6dccc522
Jul 30 17:14:12.789229 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.789190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerStarted","Data":"bb16776c0f966ebb8ceaa32fb9212560ff30939836687bf46cccf70dd3e410e5"}
Jul 30 17:14:12.789426 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:12.789274    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerStarted","Data":"a196ee5f4f5702d98104d816ba5520519d4f2ba87462d04b01700f0a6dccc522"}
Jul 30 17:14:14.375893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:14.375846    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 30 17:14:16.821199 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:16.821165    2569 generic.go:356] "Generic (PLEG): container finished" podID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerID="bb16776c0f966ebb8ceaa32fb9212560ff30939836687bf46cccf70dd3e410e5" exitCode=0
Jul 30 17:14:16.821654 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:16.821236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerDied","Data":"bb16776c0f966ebb8ceaa32fb9212560ff30939836687bf46cccf70dd3e410e5"}
Jul 30 17:14:17.827628 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.827596    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerStarted","Data":"44e3ec8087e4282752b3e448edf2fd36c84796c899e72868bcff0ae9c8b537b8"}
Jul 30 17:14:17.828004 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.827664    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerStarted","Data":"442a13028f18b2be6a0dfa0cc2aec84b4d823187e3e62687e4063d5ff438a335"}
Jul 30 17:14:17.828004 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.827689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerStarted","Data":"c80e994b3048ddf8a3c60e3f572637ec0f8263908945fc9733754237f0d2ad74"}
Jul 30 17:14:17.828004 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.827996    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:17.828122 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.828057    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:17.828122 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.828087    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:17.829417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.829392    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:14:17.830042 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.830014    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:17.849244 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:17.849200    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podStartSLOduration=2.957549061 podStartE2EDuration="6.849187224s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.891638163s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:14:17.848060322 +0000 UTC m=+825.880870651" watchObservedRunningTime="2026-07-30 17:14:17.849187224 +0000 UTC m=+825.881997560"
Jul 30 17:14:18.831857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:18.831815    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:14:18.832338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:18.832292    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:19.376027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:19.375986    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 30 17:14:19.381379 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:19.381345    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 30 17:14:19.381507 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:19.381479    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:14:19.381730 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:19.381704    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:19.381822 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:19.381807    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:14:22.061922 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.061891    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:14:22.071643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.071617    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-rbac-proxy-sar-config\") pod \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") "
Jul 30 17:14:22.071739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.071649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-cabundle-cert\") pod \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") "
Jul 30 17:14:22.071739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.071669    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqt94\" (UniqueName: \"kubernetes.io/projected/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-api-access-nqt94\") pod \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") "
Jul 30 17:14:22.071827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.071746    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kserve-provision-location\") pod \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") "
Jul 30 17:14:22.071827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.071776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-proxy-tls\") pod \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\" (UID: \"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a\") "
Jul 30 17:14:22.072041 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.072022    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" (UID: "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:14:22.072089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.072063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" (UID: "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:14:22.072132 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.072067    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" (UID: "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:14:22.073897 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.073872    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-api-access-nqt94" (OuterVolumeSpecName: "kube-api-access-nqt94") pod "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" (UID: "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a"). InnerVolumeSpecName "kube-api-access-nqt94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:14:22.073961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.073895    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" (UID: "201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:14:22.172312 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.172281    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:14:22.172312 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.172306    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:14:22.172312 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.172316    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:14:22.172530 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.172324    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:14:22.172530 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.172333    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqt94\" (UniqueName: \"kubernetes.io/projected/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a-kube-api-access-nqt94\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:14:22.848293 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.848247    2569 generic.go:356] "Generic (PLEG): container finished" podID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerID="0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa" exitCode=0
Jul 30 17:14:22.848466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.848320    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"
Jul 30 17:14:22.848466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.848331    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerDied","Data":"0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa"}
Jul 30 17:14:22.848466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.848416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7" event={"ID":"201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a","Type":"ContainerDied","Data":"685ce9fb8a3422528dc7de3d5b0cedae39308dbe9ec883d0dc34e5d597639f8f"}
Jul 30 17:14:22.848466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.848461    2569 scope.go:117] "RemoveContainer" containerID="0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa"
Jul 30 17:14:22.856174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.856155    2569 scope.go:117] "RemoveContainer" containerID="ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2"
Jul 30 17:14:22.863050 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.863030    2569 scope.go:117] "RemoveContainer" containerID="3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322"
Jul 30 17:14:22.868069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.868050    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"]
Jul 30 17:14:22.870177 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.870145    2569 scope.go:117] "RemoveContainer" containerID="3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44"
Jul 30 17:14:22.872394 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.872376    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-598dd948fc-d2zq7"]
Jul 30 17:14:22.877482 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.877467    2569 scope.go:117] "RemoveContainer" containerID="0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa"
Jul 30 17:14:22.877732 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:14:22.877713    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa\": container with ID starting with 0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa not found: ID does not exist" containerID="0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa"
Jul 30 17:14:22.877784 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.877744    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa"} err="failed to get container status \"0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa\": rpc error: code = NotFound desc = could not find container \"0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa\": container with ID starting with 0bb409f128c619b81e6825a8edb2ebbc0ea4f1fd9d3487b5e93cd4817b3aaafa not found: ID does not exist"
Jul 30 17:14:22.877784 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.877763    2569 scope.go:117] "RemoveContainer" containerID="ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2"
Jul 30 17:14:22.877976 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:14:22.877957    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2\": container with ID starting with ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2 not found: ID does not exist" containerID="ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2"
Jul 30 17:14:22.878018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.877982    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2"} err="failed to get container status \"ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2\": rpc error: code = NotFound desc = could not find container \"ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2\": container with ID starting with ff0b28a84e8ff33a48596453278a082193ae0e2e1cc16600f0190123c81316b2 not found: ID does not exist"
Jul 30 17:14:22.878018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.877998    2569 scope.go:117] "RemoveContainer" containerID="3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322"
Jul 30 17:14:22.878197 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:14:22.878179    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322\": container with ID starting with 3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322 not found: ID does not exist" containerID="3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322"
Jul 30 17:14:22.878231 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.878202    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322"} err="failed to get container status \"3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322\": rpc error: code = NotFound desc = could not find container \"3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322\": container with ID starting with 3f230a87996f9cc5effba114e4fd6f37b79edaf191ea648645ffc45cdabcc322 not found: ID does not exist"
Jul 30 17:14:22.878231 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.878217    2569 scope.go:117] "RemoveContainer" containerID="3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44"
Jul 30 17:14:22.878442 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:14:22.878426    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44\": container with ID starting with 3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44 not found: ID does not exist" containerID="3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44"
Jul 30 17:14:22.878483 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:22.878446    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44"} err="failed to get container status \"3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44\": rpc error: code = NotFound desc = could not find container \"3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44\": container with ID starting with 3e88b10f0ceb990f73a31dd37c4a39b5866f3a3b583c45303a32eda607ee7b44 not found: ID does not exist"
Jul 30 17:14:23.836643 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:23.836616    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:14:23.837177 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:23.837106    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:14:23.837472 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:23.837451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:24.651425 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:24.651392    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" path="/var/lib/kubelet/pods/201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a/volumes"
Jul 30 17:14:33.837234 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:33.837194    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:14:33.837706 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:33.837604    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:43.837312 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:43.837272    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:14:43.837779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:43.837732    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:14:53.837133 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:53.837087    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:14:53.837648 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:14:53.837611    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:15:03.837402 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:03.837362    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:15:03.837888 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:03.837858    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:15:13.837957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:13.837907    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:15:13.838468 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:13.838420    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:15:23.837880 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:23.837787    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:15:23.838472 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:23.838383    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:15:33.837462 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:33.837431    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:15:33.837957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:33.837925    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:15:47.180713 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.180675    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"]
Jul 30 17:15:47.181305 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.181024    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" containerID="cri-o://c80e994b3048ddf8a3c60e3f572637ec0f8263908945fc9733754237f0d2ad74" gracePeriod=30
Jul 30 17:15:47.181305 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.181045    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" containerID="cri-o://442a13028f18b2be6a0dfa0cc2aec84b4d823187e3e62687e4063d5ff438a335" gracePeriod=30
Jul 30 17:15:47.181305 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.181104    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" containerID="cri-o://44e3ec8087e4282752b3e448edf2fd36c84796c899e72868bcff0ae9c8b537b8" gracePeriod=30
Jul 30 17:15:47.338558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338526    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"]
Jul 30 17:15:47.338814 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338802    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338816    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338824    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338831    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338843    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="storage-initializer"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338849    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="storage-initializer"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338856    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy"
Jul 30 17:15:47.338868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338861    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy"
Jul 30 17:15:47.339105 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338909    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="agent"
Jul 30 17:15:47.339105 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338917    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kserve-container"
Jul 30 17:15:47.339105 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.338923    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="201a827d-5f5c-4cdf-a1a0-48e3cf7b3d3a" containerName="kube-rbac-proxy"
Jul 30 17:15:47.341814 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.341795    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.343998 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.343976    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 30 17:15:47.344124 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.343982    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 30 17:15:47.352935 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.352913    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"]
Jul 30 17:15:47.433338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.433310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.433537 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.433347    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f93bde-0e6a-4807-89de-ae182cc0211e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.433537 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.433373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trwfd\" (UniqueName: \"kubernetes.io/projected/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-api-access-trwfd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.433537 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.433452    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.433537 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.433526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.534497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.534464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.534695 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.534518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.534695 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.534537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f93bde-0e6a-4807-89de-ae182cc0211e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.534695 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.534565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trwfd\" (UniqueName: \"kubernetes.io/projected/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-api-access-trwfd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.534695 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:15:47.534637    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 30 17:15:47.534695 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.534673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.534972 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:15:47.534701    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls podName:a0f93bde-0e6a-4807-89de-ae182cc0211e nodeName:}" failed. No retries permitted until 2026-07-30 17:15:48.034679984 +0000 UTC m=+916.067490301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-rkc6t" (UID: "a0f93bde-0e6a-4807-89de-ae182cc0211e") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 30 17:15:47.534972 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.534960    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f93bde-0e6a-4807-89de-ae182cc0211e-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.535278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.535231    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.535352 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.535241    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:47.545884 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:47.545861    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trwfd\" (UniqueName: \"kubernetes.io/projected/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-api-access-trwfd\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:48.038587 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.038536    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:48.040985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.040950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-rkc6t\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:48.167059 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.167023    2569 generic.go:356] "Generic (PLEG): container finished" podID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerID="442a13028f18b2be6a0dfa0cc2aec84b4d823187e3e62687e4063d5ff438a335" exitCode=2
Jul 30 17:15:48.167227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.167079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerDied","Data":"442a13028f18b2be6a0dfa0cc2aec84b4d823187e3e62687e4063d5ff438a335"}
Jul 30 17:15:48.252003 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.251969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:15:48.379116 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.379080    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"]
Jul 30 17:15:48.381995 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:15:48.381966    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f93bde_0e6a_4807_89de_ae182cc0211e.slice/crio-b864fec45f3f99003eef594d8bb91d4ee2ab8a310e34d58ffdfb1ef374663ee5 WatchSource:0}: Error finding container b864fec45f3f99003eef594d8bb91d4ee2ab8a310e34d58ffdfb1ef374663ee5: Status 404 returned error can't find the container with id b864fec45f3f99003eef594d8bb91d4ee2ab8a310e34d58ffdfb1ef374663ee5
Jul 30 17:15:48.832656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:48.832609    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 30 17:15:49.172919 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:49.172828    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerStarted","Data":"b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f"}
Jul 30 17:15:49.172919 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:49.172889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerStarted","Data":"b864fec45f3f99003eef594d8bb91d4ee2ab8a310e34d58ffdfb1ef374663ee5"}
Jul 30 17:15:52.187071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:52.187038    2569 generic.go:356] "Generic (PLEG): container finished" podID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerID="c80e994b3048ddf8a3c60e3f572637ec0f8263908945fc9733754237f0d2ad74" exitCode=0
Jul 30 17:15:52.187481 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:52.187111    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerDied","Data":"c80e994b3048ddf8a3c60e3f572637ec0f8263908945fc9733754237f0d2ad74"}
Jul 30 17:15:53.832774 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:53.832728    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 30 17:15:53.837097 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:53.837066    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:15:53.838702 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:53.838676    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:15:58.832120 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:58.832069    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 30 17:15:58.832617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:15:58.832202    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:16:03.832493 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:03.832451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 30 17:16:03.837802 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:03.837776    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:16:03.838429 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:03.838399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:16:08.832063 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:08.832020    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 30 17:16:13.832748 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:13.832698    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 30 17:16:13.837050 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:13.837020    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 30 17:16:13.837184 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:13.837167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:16:13.838642 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:13.838606    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 17:16:13.838752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:13.838708    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:16:17.279952 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.279922    2569 generic.go:356] "Generic (PLEG): container finished" podID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerID="44e3ec8087e4282752b3e448edf2fd36c84796c899e72868bcff0ae9c8b537b8" exitCode=0
Jul 30 17:16:17.280369 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.280004    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerDied","Data":"44e3ec8087e4282752b3e448edf2fd36c84796c899e72868bcff0ae9c8b537b8"}
Jul 30 17:16:17.819706 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.819653    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:16:17.986817 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.986772    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-rbac-proxy-sar-config\") pod \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") "
Jul 30 17:16:17.986817 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.986818    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg5pd\" (UniqueName: \"kubernetes.io/projected/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-api-access-mg5pd\") pod \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") "
Jul 30 17:16:17.987069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.986860    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-cabundle-cert\") pod \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") "
Jul 30 17:16:17.987069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.986891    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af03e7a-542b-4636-9f8d-bf0fc11208bb-proxy-tls\") pod \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") "
Jul 30 17:16:17.987069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.986939    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kserve-provision-location\") pod \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\" (UID: \"4af03e7a-542b-4636-9f8d-bf0fc11208bb\") "
Jul 30 17:16:17.987273 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.987219    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4af03e7a-542b-4636-9f8d-bf0fc11208bb" (UID: "4af03e7a-542b-4636-9f8d-bf0fc11208bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:16:17.987406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.987302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4af03e7a-542b-4636-9f8d-bf0fc11208bb" (UID: "4af03e7a-542b-4636-9f8d-bf0fc11208bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:16:17.987406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.987317    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4af03e7a-542b-4636-9f8d-bf0fc11208bb" (UID: "4af03e7a-542b-4636-9f8d-bf0fc11208bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:16:17.988942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.988924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af03e7a-542b-4636-9f8d-bf0fc11208bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4af03e7a-542b-4636-9f8d-bf0fc11208bb" (UID: "4af03e7a-542b-4636-9f8d-bf0fc11208bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:16:17.989067 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:17.989047    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-api-access-mg5pd" (OuterVolumeSpecName: "kube-api-access-mg5pd") pod "4af03e7a-542b-4636-9f8d-bf0fc11208bb" (UID: "4af03e7a-542b-4636-9f8d-bf0fc11208bb"). InnerVolumeSpecName "kube-api-access-mg5pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:16:18.088438 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.088306    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:16:18.088438 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.088344    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4af03e7a-542b-4636-9f8d-bf0fc11208bb-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:16:18.088438 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.088354    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:16:18.088438 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.088365    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:16:18.088438 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.088373    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5pd\" (UniqueName: \"kubernetes.io/projected/4af03e7a-542b-4636-9f8d-bf0fc11208bb-kube-api-access-mg5pd\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:16:18.285701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.285666    2569 generic.go:356] "Generic (PLEG): container finished" podID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerID="b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f" exitCode=0
Jul 30 17:16:18.286115 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.285742    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerDied","Data":"b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f"}
Jul 30 17:16:18.287282 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.287248    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:16:18.288174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.288154    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg" event={"ID":"4af03e7a-542b-4636-9f8d-bf0fc11208bb","Type":"ContainerDied","Data":"a196ee5f4f5702d98104d816ba5520519d4f2ba87462d04b01700f0a6dccc522"}
Jul 30 17:16:18.288285 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.288232    2569 scope.go:117] "RemoveContainer" containerID="44e3ec8087e4282752b3e448edf2fd36c84796c899e72868bcff0ae9c8b537b8"
Jul 30 17:16:18.288366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.288232    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"
Jul 30 17:16:18.296482 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.296441    2569 scope.go:117] "RemoveContainer" containerID="442a13028f18b2be6a0dfa0cc2aec84b4d823187e3e62687e4063d5ff438a335"
Jul 30 17:16:18.303798 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.303777    2569 scope.go:117] "RemoveContainer" containerID="c80e994b3048ddf8a3c60e3f572637ec0f8263908945fc9733754237f0d2ad74"
Jul 30 17:16:18.312777 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.312759    2569 scope.go:117] "RemoveContainer" containerID="bb16776c0f966ebb8ceaa32fb9212560ff30939836687bf46cccf70dd3e410e5"
Jul 30 17:16:18.319870 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.319840    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"]
Jul 30 17:16:18.326195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.326173    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-d7459ccb5-lckrg"]
Jul 30 17:16:18.651587 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:16:18.651556    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" path="/var/lib/kubelet/pods/4af03e7a-542b-4636-9f8d-bf0fc11208bb/volumes"
Jul 30 17:18:16.786609 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:16.786567    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerStarted","Data":"36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca"}
Jul 30 17:18:16.787087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:16.786633    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerStarted","Data":"9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2"}
Jul 30 17:18:16.787087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:16.786658    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:18:16.787087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:16.786689    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:18:16.788018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:16.787992    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 30 17:18:16.811291 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:16.811233    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" podStartSLOduration=2.9066431120000003 podStartE2EDuration="2m29.811219464s" totalImagesPullingTime="1m57.795456003s" totalInitContainerRuntime="29.109120349s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:18:16.809316083 +0000 UTC m=+1064.842126421" watchObservedRunningTime="2026-07-30 17:18:16.811219464 +0000 UTC m=+1064.844029800"
Jul 30 17:18:17.790663 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:17.790621    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 30 17:18:22.795637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:22.795602    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:18:22.796180 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:22.796155    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 30 17:18:32.797290 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:32.797191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:18:43.526055 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:43.526021    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"]
Jul 30 17:18:43.526529 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:43.526275    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container" containerID="cri-o://9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2" gracePeriod=30
Jul 30 17:18:43.526529 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:43.526339    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kube-rbac-proxy" containerID="cri-o://36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca" gracePeriod=30
Jul 30 17:18:43.892477 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:43.892392    2569 generic.go:356] "Generic (PLEG): container finished" podID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerID="36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca" exitCode=2
Jul 30 17:18:43.892477 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:43.892438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerDied","Data":"36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca"}
Jul 30 17:18:44.889418 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.889389    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:18:44.897843 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.897820    2569 generic.go:356] "Generic (PLEG): container finished" podID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerID="9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2" exitCode=0
Jul 30 17:18:44.897949 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.897869    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"
Jul 30 17:18:44.897949 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.897894    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerDied","Data":"9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2"}
Jul 30 17:18:44.898022 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.897950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t" event={"ID":"a0f93bde-0e6a-4807-89de-ae182cc0211e","Type":"ContainerDied","Data":"b864fec45f3f99003eef594d8bb91d4ee2ab8a310e34d58ffdfb1ef374663ee5"}
Jul 30 17:18:44.898022 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.897975    2569 scope.go:117] "RemoveContainer" containerID="36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca"
Jul 30 17:18:44.905218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.905201    2569 scope.go:117] "RemoveContainer" containerID="9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2"
Jul 30 17:18:44.913021 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.913002    2569 scope.go:117] "RemoveContainer" containerID="b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f"
Jul 30 17:18:44.921078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.921050    2569 scope.go:117] "RemoveContainer" containerID="36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca"
Jul 30 17:18:44.921376 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:18:44.921348    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca\": container with ID starting with 36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca not found: ID does not exist" containerID="36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca"
Jul 30 17:18:44.921466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.921377    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca"} err="failed to get container status \"36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca\": rpc error: code = NotFound desc = could not find container \"36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca\": container with ID starting with 36045d6de90bb6e7c1ad859f5f93132482bcd53bae35d0544682b6ffe8893cca not found: ID does not exist"
Jul 30 17:18:44.921466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.921394    2569 scope.go:117] "RemoveContainer" containerID="9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2"
Jul 30 17:18:44.921645 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:18:44.921614    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2\": container with ID starting with 9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2 not found: ID does not exist" containerID="9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2"
Jul 30 17:18:44.921752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.921651    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2"} err="failed to get container status \"9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2\": rpc error: code = NotFound desc = could not find container \"9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2\": container with ID starting with 9f028c52343d173d0052fd5977f0aea1f0320aa5106145f3add78ca5d746c2d2 not found: ID does not exist"
Jul 30 17:18:44.921752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.921667    2569 scope.go:117] "RemoveContainer" containerID="b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f"
Jul 30 17:18:44.921947 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:18:44.921919    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f\": container with ID starting with b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f not found: ID does not exist" containerID="b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f"
Jul 30 17:18:44.921997 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:44.921978    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f"} err="failed to get container status \"b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f\": rpc error: code = NotFound desc = could not find container \"b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f\": container with ID starting with b44f59a603b577bd62bba682887f1ed667965509e3031014af04c04f9308297f not found: ID does not exist"
Jul 30 17:18:45.015862 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.015830    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-rbac-proxy-sar-config\") pod \"a0f93bde-0e6a-4807-89de-ae182cc0211e\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") "
Jul 30 17:18:45.016024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.015884    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls\") pod \"a0f93bde-0e6a-4807-89de-ae182cc0211e\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") "
Jul 30 17:18:45.016024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.015927    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trwfd\" (UniqueName: \"kubernetes.io/projected/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-api-access-trwfd\") pod \"a0f93bde-0e6a-4807-89de-ae182cc0211e\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") "
Jul 30 17:18:45.016024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.015949    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-cabundle-cert\") pod \"a0f93bde-0e6a-4807-89de-ae182cc0211e\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") "
Jul 30 17:18:45.016135 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.015998    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f93bde-0e6a-4807-89de-ae182cc0211e-kserve-provision-location\") pod \"a0f93bde-0e6a-4807-89de-ae182cc0211e\" (UID: \"a0f93bde-0e6a-4807-89de-ae182cc0211e\") "
Jul 30 17:18:45.016378 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.016347    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0f93bde-0e6a-4807-89de-ae182cc0211e" (UID: "a0f93bde-0e6a-4807-89de-ae182cc0211e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:18:45.016462 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.016359    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0f93bde-0e6a-4807-89de-ae182cc0211e" (UID: "a0f93bde-0e6a-4807-89de-ae182cc0211e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:18:45.018093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.018072    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-api-access-trwfd" (OuterVolumeSpecName: "kube-api-access-trwfd") pod "a0f93bde-0e6a-4807-89de-ae182cc0211e" (UID: "a0f93bde-0e6a-4807-89de-ae182cc0211e"). InnerVolumeSpecName "kube-api-access-trwfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:18:45.018171 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.018140    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0f93bde-0e6a-4807-89de-ae182cc0211e" (UID: "a0f93bde-0e6a-4807-89de-ae182cc0211e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:18:45.020687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.020665    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f93bde-0e6a-4807-89de-ae182cc0211e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0f93bde-0e6a-4807-89de-ae182cc0211e" (UID: "a0f93bde-0e6a-4807-89de-ae182cc0211e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:18:45.116976 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.116939    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0f93bde-0e6a-4807-89de-ae182cc0211e-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:18:45.116976 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.116969    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:18:45.116976 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.116979    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f93bde-0e6a-4807-89de-ae182cc0211e-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:18:45.117205 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.116988    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trwfd\" (UniqueName: \"kubernetes.io/projected/a0f93bde-0e6a-4807-89de-ae182cc0211e-kube-api-access-trwfd\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:18:45.117205 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.116999    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0f93bde-0e6a-4807-89de-ae182cc0211e-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:18:45.220820 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.220793    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"]
Jul 30 17:18:45.227604 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:45.227580    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-rkc6t"]
Jul 30 17:18:46.651827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:46.651794    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" path="/var/lib/kubelet/pods/a0f93bde-0e6a-4807-89de-ae182cc0211e/volumes"
Jul 30 17:18:48.671109 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671069    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"]
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671452    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="storage-initializer"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671470    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="storage-initializer"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671488    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kube-rbac-proxy"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671496    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kube-rbac-proxy"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671517    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671525    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671536    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671543    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671554    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671562    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671576    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="storage-initializer"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671585    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="storage-initializer"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671595    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671603    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671667    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="agent"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671679    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kserve-container"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671688    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f93bde-0e6a-4807-89de-ae182cc0211e" containerName="kube-rbac-proxy"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671698    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kserve-container"
Jul 30 17:18:48.671724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.671707    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af03e7a-542b-4636-9f8d-bf0fc11208bb" containerName="kube-rbac-proxy"
Jul 30 17:18:48.709973 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.709937    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"]
Jul 30 17:18:48.710160 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.710074    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.712844 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.712817    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 30 17:18:48.712975 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.712857    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:18:48.712975 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.712941    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 30 17:18:48.713957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.713932    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:18:48.714087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.713990    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:18:48.714087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.714002    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:18:48.714198 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.714139    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:18:48.845586 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.845548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.845768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.845597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fgqh\" (UniqueName: \"kubernetes.io/projected/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-api-access-4fgqh\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.845768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.845717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc899d05-1168-4a75-872a-00c8050ab9f3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.845768 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.845753    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.845923 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.845776    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc899d05-1168-4a75-872a-00c8050ab9f3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946165 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fgqh\" (UniqueName: \"kubernetes.io/projected/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-api-access-4fgqh\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc899d05-1168-4a75-872a-00c8050ab9f3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc899d05-1168-4a75-872a-00c8050ab9f3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946277    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946660 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946639    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc899d05-1168-4a75-872a-00c8050ab9f3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.946924 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.946904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.947021 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.947004    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.948846 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.948828    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc899d05-1168-4a75-872a-00c8050ab9f3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:48.956183 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:48.956154    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fgqh\" (UniqueName: \"kubernetes.io/projected/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-api-access-4fgqh\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-cwzpc\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:49.020913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:49.020883    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:18:49.164043 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:49.164013    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"]
Jul 30 17:18:49.167306 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:18:49.167277    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc899d05_1168_4a75_872a_00c8050ab9f3.slice/crio-49525b6ad859999b1fd24a561cd174a605523e3465cb370c94b935a9b97f0069 WatchSource:0}: Error finding container 49525b6ad859999b1fd24a561cd174a605523e3465cb370c94b935a9b97f0069: Status 404 returned error can't find the container with id 49525b6ad859999b1fd24a561cd174a605523e3465cb370c94b935a9b97f0069
Jul 30 17:18:49.917646 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:49.917599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerStarted","Data":"4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88"}
Jul 30 17:18:49.918040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:18:49.917669    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerStarted","Data":"49525b6ad859999b1fd24a561cd174a605523e3465cb370c94b935a9b97f0069"}
Jul 30 17:19:17.020534 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:17.020495    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerID="4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88" exitCode=0
Jul 30 17:19:17.020927 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:17.020569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerDied","Data":"4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88"}
Jul 30 17:19:18.027130 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:18.027093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerStarted","Data":"1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055"}
Jul 30 17:19:18.027573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:18.027153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerStarted","Data":"85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5"}
Jul 30 17:19:18.027573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:18.027411    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:19:18.027573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:18.027474    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:19:18.046403 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:18.046343    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" podStartSLOduration=2.7373332230000003 podStartE2EDuration="30.046326643s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.30899342s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:19:18.045872016 +0000 UTC m=+1126.078682355" watchObservedRunningTime="2026-07-30 17:19:18.046326643 +0000 UTC m=+1126.079136979"
Jul 30 17:19:24.036220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:19:24.036190    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:20:55.650746 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:20:55.650706    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:21:04.961928 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:04.961883    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"]
Jul 30 17:21:04.962430 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:04.962137    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kserve-container" containerID="cri-o://85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5" gracePeriod=30
Jul 30 17:21:04.962430 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:04.962198    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kube-rbac-proxy" containerID="cri-o://1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055" gracePeriod=30
Jul 30 17:21:05.421165 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:05.421069    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerID="1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055" exitCode=2
Jul 30 17:21:05.421165 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:05.421128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerDied","Data":"1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055"}
Jul 30 17:21:05.648278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:05.648199    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.24:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 30 17:21:06.292268 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.292228    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:21:06.381881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.381849    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fgqh\" (UniqueName: \"kubernetes.io/projected/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-api-access-4fgqh\") pod \"fc899d05-1168-4a75-872a-00c8050ab9f3\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") "
Jul 30 17:21:06.381881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.381889    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-rbac-proxy-sar-config\") pod \"fc899d05-1168-4a75-872a-00c8050ab9f3\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") "
Jul 30 17:21:06.382141 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.381911    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-cabundle-cert\") pod \"fc899d05-1168-4a75-872a-00c8050ab9f3\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") "
Jul 30 17:21:06.382141 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.381938    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc899d05-1168-4a75-872a-00c8050ab9f3-kserve-provision-location\") pod \"fc899d05-1168-4a75-872a-00c8050ab9f3\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") "
Jul 30 17:21:06.382141 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.381995    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc899d05-1168-4a75-872a-00c8050ab9f3-proxy-tls\") pod \"fc899d05-1168-4a75-872a-00c8050ab9f3\" (UID: \"fc899d05-1168-4a75-872a-00c8050ab9f3\") "
Jul 30 17:21:06.382358 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.382331    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc899d05-1168-4a75-872a-00c8050ab9f3" (UID: "fc899d05-1168-4a75-872a-00c8050ab9f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:21:06.382412 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.382334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc899d05-1168-4a75-872a-00c8050ab9f3" (UID: "fc899d05-1168-4a75-872a-00c8050ab9f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:21:06.384057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.384026    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc899d05-1168-4a75-872a-00c8050ab9f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc899d05-1168-4a75-872a-00c8050ab9f3" (UID: "fc899d05-1168-4a75-872a-00c8050ab9f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:21:06.384057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.384031    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-api-access-4fgqh" (OuterVolumeSpecName: "kube-api-access-4fgqh") pod "fc899d05-1168-4a75-872a-00c8050ab9f3" (UID: "fc899d05-1168-4a75-872a-00c8050ab9f3"). InnerVolumeSpecName "kube-api-access-4fgqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:21:06.388247 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.388221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc899d05-1168-4a75-872a-00c8050ab9f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc899d05-1168-4a75-872a-00c8050ab9f3" (UID: "fc899d05-1168-4a75-872a-00c8050ab9f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:21:06.426923 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.426838    2569 generic.go:356] "Generic (PLEG): container finished" podID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerID="85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5" exitCode=0
Jul 30 17:21:06.427074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.426923    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"
Jul 30 17:21:06.427074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.426924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerDied","Data":"85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5"}
Jul 30 17:21:06.427179 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.427081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc" event={"ID":"fc899d05-1168-4a75-872a-00c8050ab9f3","Type":"ContainerDied","Data":"49525b6ad859999b1fd24a561cd174a605523e3465cb370c94b935a9b97f0069"}
Jul 30 17:21:06.427179 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.427111    2569 scope.go:117] "RemoveContainer" containerID="1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055"
Jul 30 17:21:06.435641 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.435583    2569 scope.go:117] "RemoveContainer" containerID="85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5"
Jul 30 17:21:06.442705 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.442685    2569 scope.go:117] "RemoveContainer" containerID="4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88"
Jul 30 17:21:06.449388 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.449364    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"]
Jul 30 17:21:06.449764 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.449749    2569 scope.go:117] "RemoveContainer" containerID="1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055"
Jul 30 17:21:06.449986 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:21:06.449965    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055\": container with ID starting with 1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055 not found: ID does not exist" containerID="1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055"
Jul 30 17:21:06.450048 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.449998    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055"} err="failed to get container status \"1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055\": rpc error: code = NotFound desc = could not find container \"1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055\": container with ID starting with 1207fb775ac272ec80fbd59cbee1c85ad7d11f2007ba49ad96063c613fa11055 not found: ID does not exist"
Jul 30 17:21:06.450048 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.450025    2569 scope.go:117] "RemoveContainer" containerID="85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5"
Jul 30 17:21:06.450362 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:21:06.450342    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5\": container with ID starting with 85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5 not found: ID does not exist" containerID="85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5"
Jul 30 17:21:06.450409 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.450369    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5"} err="failed to get container status \"85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5\": rpc error: code = NotFound desc = could not find container \"85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5\": container with ID starting with 85249757986dffc61ebdf029c38ec3f90bbc456450d68e35a046507ede608dc5 not found: ID does not exist"
Jul 30 17:21:06.450409 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.450384    2569 scope.go:117] "RemoveContainer" containerID="4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88"
Jul 30 17:21:06.450598 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:21:06.450579    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88\": container with ID starting with 4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88 not found: ID does not exist" containerID="4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88"
Jul 30 17:21:06.450642 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.450604    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88"} err="failed to get container status \"4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88\": rpc error: code = NotFound desc = could not find container \"4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88\": container with ID starting with 4e2b5fe80aa0dd37f702fc4b22e65a704f18f107861f89cf95978197207a9d88 not found: ID does not exist"
Jul 30 17:21:06.454467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.454446    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-cwzpc"]
Jul 30 17:21:06.482573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.482549    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:21:06.482573 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.482572    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc899d05-1168-4a75-872a-00c8050ab9f3-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:21:06.482701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.482584    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc899d05-1168-4a75-872a-00c8050ab9f3-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:21:06.482701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.482592    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc899d05-1168-4a75-872a-00c8050ab9f3-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:21:06.482701 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.482602    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fgqh\" (UniqueName: \"kubernetes.io/projected/fc899d05-1168-4a75-872a-00c8050ab9f3-kube-api-access-4fgqh\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:21:06.651833 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:06.651803    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" path="/var/lib/kubelet/pods/fc899d05-1168-4a75-872a-00c8050ab9f3/volumes"
Jul 30 17:21:10.120039 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120005    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"]
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120281    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kserve-container"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120296    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kserve-container"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120314    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="storage-initializer"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120320    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="storage-initializer"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120327    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kube-rbac-proxy"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120333    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kube-rbac-proxy"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120377    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kserve-container"
Jul 30 17:21:10.120471 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.120388    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc899d05-1168-4a75-872a-00c8050ab9f3" containerName="kube-rbac-proxy"
Jul 30 17:21:10.123518 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.123498    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.126161 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.126135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:21:10.126288 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.126184    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 30 17:21:10.126288 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.126196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:21:10.127392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.127374    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:21:10.127392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.127388    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:21:10.127653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.127638    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:21:10.127721 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.127685    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 30 17:21:10.133366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.133345    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"]
Jul 30 17:21:10.205032 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.204996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.205200 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.205048    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.205200 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.205081    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.205200 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.205105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86bm\" (UniqueName: \"kubernetes.io/projected/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-api-access-z86bm\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.205200 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.205162    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.306434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.306399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.306434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.306440    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.306656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.306479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.306656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.306498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.306656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.306524    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86bm\" (UniqueName: \"kubernetes.io/projected/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-api-access-z86bm\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.306978 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.306954    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.307154 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.307133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.307238 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.307183    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.309032 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.309008    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.315095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.315072    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86bm\" (UniqueName: \"kubernetes.io/projected/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-api-access-z86bm\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.434342 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.434309    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:10.562547 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:10.562511    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"]
Jul 30 17:21:10.565387 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:21:10.565355    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7b28be_3c91_4354_80f0_a4113a3c3ec7.slice/crio-5687088a98585a1f3d5e44caa7481087cb67cb4c6b475a5a94854886e0aa1cce WatchSource:0}: Error finding container 5687088a98585a1f3d5e44caa7481087cb67cb4c6b475a5a94854886e0aa1cce: Status 404 returned error can't find the container with id 5687088a98585a1f3d5e44caa7481087cb67cb4c6b475a5a94854886e0aa1cce
Jul 30 17:21:11.447558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:11.447520    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerStarted","Data":"f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124"}
Jul 30 17:21:11.447954 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:11.447584    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerStarted","Data":"5687088a98585a1f3d5e44caa7481087cb67cb4c6b475a5a94854886e0aa1cce"}
Jul 30 17:21:31.520027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:31.519991    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerID="f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124" exitCode=0
Jul 30 17:21:31.520565 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:31.520070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerDied","Data":"f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124"}
Jul 30 17:21:32.526893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:32.526382    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerStarted","Data":"bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd"}
Jul 30 17:21:32.526893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:32.526451    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerStarted","Data":"a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c"}
Jul 30 17:21:32.526893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:32.526644    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:32.548026 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:32.547974    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" podStartSLOduration=2.099991426 podStartE2EDuration="22.547953351s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.447961925s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:21:32.545516292 +0000 UTC m=+1260.578326639" watchObservedRunningTime="2026-07-30 17:21:32.547953351 +0000 UTC m=+1260.580763688"
Jul 30 17:21:33.530853 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:33.530821    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:21:39.540807 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:21:39.540776    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:23:09.651764 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:09.651672    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:23:16.769068 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:16.769034    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"]
Jul 30 17:23:16.769558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:16.769288    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kserve-container" containerID="cri-o://a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c" gracePeriod=30
Jul 30 17:23:16.769558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:16.769345    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kube-rbac-proxy" containerID="cri-o://bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd" gracePeriod=30
Jul 30 17:23:16.909800 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:16.909767    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerID="bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd" exitCode=2
Jul 30 17:23:16.909958 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:16.909840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerDied","Data":"bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd"}
Jul 30 17:23:18.120952 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.120918    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:23:18.307481 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307389    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-cabundle-cert\") pod \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") "
Jul 30 17:23:18.307481 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307423    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kserve-provision-location\") pod \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") "
Jul 30 17:23:18.307481 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307466    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-proxy-tls\") pod \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") "
Jul 30 17:23:18.307481 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-rbac-proxy-sar-config\") pod \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") "
Jul 30 17:23:18.307841 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307526    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z86bm\" (UniqueName: \"kubernetes.io/projected/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-api-access-z86bm\") pod \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\" (UID: \"6a7b28be-3c91-4354-80f0-a4113a3c3ec7\") "
Jul 30 17:23:18.307904 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307860    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a7b28be-3c91-4354-80f0-a4113a3c3ec7" (UID: "6a7b28be-3c91-4354-80f0-a4113a3c3ec7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:23:18.307957 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.307898    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a7b28be-3c91-4354-80f0-a4113a3c3ec7" (UID: "6a7b28be-3c91-4354-80f0-a4113a3c3ec7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:23:18.309638 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.309603    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a7b28be-3c91-4354-80f0-a4113a3c3ec7" (UID: "6a7b28be-3c91-4354-80f0-a4113a3c3ec7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:23:18.309638 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.309605    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-api-access-z86bm" (OuterVolumeSpecName: "kube-api-access-z86bm") pod "6a7b28be-3c91-4354-80f0-a4113a3c3ec7" (UID: "6a7b28be-3c91-4354-80f0-a4113a3c3ec7"). InnerVolumeSpecName "kube-api-access-z86bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:23:18.312362 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.312342    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a7b28be-3c91-4354-80f0-a4113a3c3ec7" (UID: "6a7b28be-3c91-4354-80f0-a4113a3c3ec7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:23:18.408320 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.408290    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:23:18.408320 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.408315    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:23:18.408320 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.408328    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:23:18.408544 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.408337    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:23:18.408544 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.408345    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z86bm\" (UniqueName: \"kubernetes.io/projected/6a7b28be-3c91-4354-80f0-a4113a3c3ec7-kube-api-access-z86bm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:23:18.923624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.923530    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerID="a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c" exitCode=0
Jul 30 17:23:18.923624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.923585    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerDied","Data":"a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c"}
Jul 30 17:23:18.923624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.923607    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"
Jul 30 17:23:18.923897 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.923655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9" event={"ID":"6a7b28be-3c91-4354-80f0-a4113a3c3ec7","Type":"ContainerDied","Data":"5687088a98585a1f3d5e44caa7481087cb67cb4c6b475a5a94854886e0aa1cce"}
Jul 30 17:23:18.923897 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.923680    2569 scope.go:117] "RemoveContainer" containerID="bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd"
Jul 30 17:23:18.931994 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.931974    2569 scope.go:117] "RemoveContainer" containerID="a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c"
Jul 30 17:23:18.938962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.938943    2569 scope.go:117] "RemoveContainer" containerID="f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124"
Jul 30 17:23:18.944407 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.944382    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"]
Jul 30 17:23:18.946570 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.946542    2569 scope.go:117] "RemoveContainer" containerID="bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd"
Jul 30 17:23:18.946872 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:23:18.946849    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd\": container with ID starting with bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd not found: ID does not exist" containerID="bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd"
Jul 30 17:23:18.946961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.946880    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd"} err="failed to get container status \"bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd\": rpc error: code = NotFound desc = could not find container \"bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd\": container with ID starting with bb96c8dba583aa2090d19801d82897634d7977def4832496dbb5ccb0df6aecdd not found: ID does not exist"
Jul 30 17:23:18.946961 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.946900    2569 scope.go:117] "RemoveContainer" containerID="a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c"
Jul 30 17:23:18.947321 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:23:18.947289    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c\": container with ID starting with a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c not found: ID does not exist" containerID="a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c"
Jul 30 17:23:18.947440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.947333    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c"} err="failed to get container status \"a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c\": rpc error: code = NotFound desc = could not find container \"a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c\": container with ID starting with a2ab0e32efd5ea86471eaf56fee9902b4ec834246bc6fd21f28e63a34db1e50c not found: ID does not exist"
Jul 30 17:23:18.947440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.947359    2569 scope.go:117] "RemoveContainer" containerID="f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124"
Jul 30 17:23:18.947763 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:23:18.947733    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124\": container with ID starting with f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124 not found: ID does not exist" containerID="f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124"
Jul 30 17:23:18.947864 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.947768    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124"} err="failed to get container status \"f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124\": rpc error: code = NotFound desc = could not find container \"f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124\": container with ID starting with f69943bdb64e97352004eacdb7d333050cbb05ffe747c4990d7a93cc4ed32124 not found: ID does not exist"
Jul 30 17:23:18.948928 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:18.948909    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-hzxq9"]
Jul 30 17:23:20.651549 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:20.651516    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" path="/var/lib/kubelet/pods/6a7b28be-3c91-4354-80f0-a4113a3c3ec7/volumes"
Jul 30 17:23:21.946778 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.946744    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"]
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947017    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kube-rbac-proxy"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947027    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kube-rbac-proxy"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947045    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="storage-initializer"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947051    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="storage-initializer"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947059    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kserve-container"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947064    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kserve-container"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947101    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kserve-container"
Jul 30 17:23:21.947147 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.947108    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7b28be-3c91-4354-80f0-a4113a3c3ec7" containerName="kube-rbac-proxy"
Jul 30 17:23:21.951904 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.951886    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:21.954448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.954415    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 30 17:23:21.954595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.954461    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:23:21.954595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.954470    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 30 17:23:21.954595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.954423    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:23:21.954595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.954463    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:23:21.954595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.954423    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:23:21.955482 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.955464    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:23:21.962620 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:21.962601    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"]
Jul 30 17:23:22.138160 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.138124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.138389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.138201    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.138389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.138236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.138389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.138292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.138389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.138310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5fm\" (UniqueName: \"kubernetes.io/projected/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-api-access-xj5fm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.239619 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.239528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.239619 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.239568    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.239619 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.239597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.239619 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.239615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5fm\" (UniqueName: \"kubernetes.io/projected/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-api-access-xj5fm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.239940 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.239643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.239940 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:23:22.239886    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 30 17:23:22.240053 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.239969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.240053 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:23:22.239975    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls podName:33632e0a-5df3-4fef-b7a1-06d15131b5a3 nodeName:}" failed. No retries permitted until 2026-07-30 17:23:22.739954023 +0000 UTC m=+1370.772764338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" (UID: "33632e0a-5df3-4fef-b7a1-06d15131b5a3") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Jul 30 17:23:22.240387 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.240366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.240476 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.240371    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.249196 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.249174    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5fm\" (UniqueName: \"kubernetes.io/projected/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-api-access-xj5fm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.744416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.744383    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.746723 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.746692    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.862112 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.862077    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:22.985151 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:22.985124    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"]
Jul 30 17:23:22.987335 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:23:22.987303    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33632e0a_5df3_4fef_b7a1_06d15131b5a3.slice/crio-b99461018d6dd1b22ccc93472d1d13c8c4a7b8d9b6fdd30ffe1d8093fe62b4d4 WatchSource:0}: Error finding container b99461018d6dd1b22ccc93472d1d13c8c4a7b8d9b6fdd30ffe1d8093fe62b4d4: Status 404 returned error can't find the container with id b99461018d6dd1b22ccc93472d1d13c8c4a7b8d9b6fdd30ffe1d8093fe62b4d4
Jul 30 17:23:23.944378 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:23.944341    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerStarted","Data":"1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f"}
Jul 30 17:23:23.944595 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:23.944412    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerStarted","Data":"b99461018d6dd1b22ccc93472d1d13c8c4a7b8d9b6fdd30ffe1d8093fe62b4d4"}
Jul 30 17:23:27.960035 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:27.959942    2569 generic.go:356] "Generic (PLEG): container finished" podID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerID="1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f" exitCode=0
Jul 30 17:23:27.960035 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:27.959985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerDied","Data":"1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f"}
Jul 30 17:23:28.965979 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:28.965938    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerStarted","Data":"cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338"}
Jul 30 17:23:28.966379 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:28.966022    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerStarted","Data":"e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1"}
Jul 30 17:23:28.966379 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:28.966243    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:28.986827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:28.986783    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" podStartSLOduration=3.339826287 podStartE2EDuration="7.986767749s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.646941462s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:23:28.986549524 +0000 UTC m=+1377.019359862" watchObservedRunningTime="2026-07-30 17:23:28.986767749 +0000 UTC m=+1377.019578086"
Jul 30 17:23:29.970140 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:29.970107    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:23:35.980065 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:23:35.980035    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:24:59.651415 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:24:59.651380    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:25:08.219249 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:08.219210    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"]
Jul 30 17:25:08.219748 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:08.219468    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kserve-container" containerID="cri-o://e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1" gracePeriod=30
Jul 30 17:25:08.219748 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:08.219538    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kube-rbac-proxy" containerID="cri-o://cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338" gracePeriod=30
Jul 30 17:25:08.348558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:08.348522    2569 generic.go:356] "Generic (PLEG): container finished" podID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerID="cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338" exitCode=2
Jul 30 17:25:08.348678 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:08.348563    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerDied","Data":"cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338"}
Jul 30 17:25:09.559381 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.559356    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:25:09.660218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660120    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kserve-provision-location\") pod \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") "
Jul 30 17:25:09.660218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660182    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-cabundle-cert\") pod \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") "
Jul 30 17:25:09.660218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660201    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj5fm\" (UniqueName: \"kubernetes.io/projected/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-api-access-xj5fm\") pod \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") "
Jul 30 17:25:09.660218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660221    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-rbac-proxy-sar-config\") pod \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") "
Jul 30 17:25:09.660594 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660292    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls\") pod \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\" (UID: \"33632e0a-5df3-4fef-b7a1-06d15131b5a3\") "
Jul 30 17:25:09.660686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660655    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33632e0a-5df3-4fef-b7a1-06d15131b5a3" (UID: "33632e0a-5df3-4fef-b7a1-06d15131b5a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:25:09.660746 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.660663    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33632e0a-5df3-4fef-b7a1-06d15131b5a3" (UID: "33632e0a-5df3-4fef-b7a1-06d15131b5a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:25:09.662414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.662379    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-api-access-xj5fm" (OuterVolumeSpecName: "kube-api-access-xj5fm") pod "33632e0a-5df3-4fef-b7a1-06d15131b5a3" (UID: "33632e0a-5df3-4fef-b7a1-06d15131b5a3"). InnerVolumeSpecName "kube-api-access-xj5fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:25:09.662521 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.662499    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33632e0a-5df3-4fef-b7a1-06d15131b5a3" (UID: "33632e0a-5df3-4fef-b7a1-06d15131b5a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:25:09.664636 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.664616    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33632e0a-5df3-4fef-b7a1-06d15131b5a3" (UID: "33632e0a-5df3-4fef-b7a1-06d15131b5a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:25:09.761175 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.761140    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:09.761175 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.761169    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj5fm\" (UniqueName: \"kubernetes.io/projected/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-api-access-xj5fm\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:09.761175 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.761180    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:09.761436 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.761189    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33632e0a-5df3-4fef-b7a1-06d15131b5a3-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:09.761436 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:09.761200    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33632e0a-5df3-4fef-b7a1-06d15131b5a3-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:10.358665 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.358633    2569 generic.go:356] "Generic (PLEG): container finished" podID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerID="e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1" exitCode=0
Jul 30 17:25:10.358839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.358700    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"
Jul 30 17:25:10.358839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.358720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerDied","Data":"e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1"}
Jul 30 17:25:10.358839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.358795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb" event={"ID":"33632e0a-5df3-4fef-b7a1-06d15131b5a3","Type":"ContainerDied","Data":"b99461018d6dd1b22ccc93472d1d13c8c4a7b8d9b6fdd30ffe1d8093fe62b4d4"}
Jul 30 17:25:10.358949 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.358837    2569 scope.go:117] "RemoveContainer" containerID="cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338"
Jul 30 17:25:10.367809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.367431    2569 scope.go:117] "RemoveContainer" containerID="e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1"
Jul 30 17:25:10.375104 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.375086    2569 scope.go:117] "RemoveContainer" containerID="1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f"
Jul 30 17:25:10.381748 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.381731    2569 scope.go:117] "RemoveContainer" containerID="cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338"
Jul 30 17:25:10.382020 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:25:10.381995    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338\": container with ID starting with cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338 not found: ID does not exist" containerID="cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338"
Jul 30 17:25:10.382069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.382029    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338"} err="failed to get container status \"cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338\": rpc error: code = NotFound desc = could not find container \"cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338\": container with ID starting with cea2d9e0b3308f20a4d58d6d44ff49429d3830949da4787a06ce382e70780338 not found: ID does not exist"
Jul 30 17:25:10.382069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.382048    2569 scope.go:117] "RemoveContainer" containerID="e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1"
Jul 30 17:25:10.382312 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:25:10.382289    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1\": container with ID starting with e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1 not found: ID does not exist" containerID="e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1"
Jul 30 17:25:10.382415 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.382321    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1"} err="failed to get container status \"e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1\": rpc error: code = NotFound desc = could not find container \"e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1\": container with ID starting with e2fad152514dd9e3d0185e00c733e67149bdcbc045f93d47fd7129d074cebec1 not found: ID does not exist"
Jul 30 17:25:10.382415 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.382339    2569 scope.go:117] "RemoveContainer" containerID="1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f"
Jul 30 17:25:10.382588 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:25:10.382570    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f\": container with ID starting with 1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f not found: ID does not exist" containerID="1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f"
Jul 30 17:25:10.382650 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.382596    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f"} err="failed to get container status \"1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f\": rpc error: code = NotFound desc = could not find container \"1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f\": container with ID starting with 1701481ee383ce24de05a3cf55aac524e464d40edee017bc931e98bd1eb2714f not found: ID does not exist"
Jul 30 17:25:10.385094 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.385069    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"]
Jul 30 17:25:10.388072 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.388050    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4zclb"]
Jul 30 17:25:10.651827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:10.651745    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" path="/var/lib/kubelet/pods/33632e0a-5df3-4fef-b7a1-06d15131b5a3/volumes"
Jul 30 17:25:13.389027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.388992    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"]
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389242    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kserve-container"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389265    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kserve-container"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389276    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="storage-initializer"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389282    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="storage-initializer"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389296    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kube-rbac-proxy"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389302    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kube-rbac-proxy"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389340    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kserve-container"
Jul 30 17:25:13.389420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.389347    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="33632e0a-5df3-4fef-b7a1-06d15131b5a3" containerName="kube-rbac-proxy"
Jul 30 17:25:13.393893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.393872    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.396215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.396194    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 30 17:25:13.396363 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.396193    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:25:13.396414 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.396376    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:25:13.396579 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.396564    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 30 17:25:13.397332 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.397317    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:25:13.397417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.397339    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:25:13.397417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.397338    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:25:13.405775 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.405754    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"]
Jul 30 17:25:13.486248 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.486218    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.486413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.486273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afb416-c271-4c66-bd80-495703396fb6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.486413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.486335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ktx\" (UniqueName: \"kubernetes.io/projected/36afb416-c271-4c66-bd80-495703396fb6-kube-api-access-82ktx\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.486413 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.486375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.486538 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.486412    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36afb416-c271-4c66-bd80-495703396fb6-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ktx\" (UniqueName: \"kubernetes.io/projected/36afb416-c271-4c66-bd80-495703396fb6-kube-api-access-82ktx\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36afb416-c271-4c66-bd80-495703396fb6-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587181    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afb416-c271-4c66-bd80-495703396fb6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587627 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afb416-c271-4c66-bd80-495703396fb6-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587921    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.587995 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.587944    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.589367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.589349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36afb416-c271-4c66-bd80-495703396fb6-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.599356 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.599318    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ktx\" (UniqueName: \"kubernetes.io/projected/36afb416-c271-4c66-bd80-495703396fb6-kube-api-access-82ktx\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.704628 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.704592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:13.831770 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:13.831738    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"]
Jul 30 17:25:13.835372 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:25:13.835345    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36afb416_c271_4c66_bd80_495703396fb6.slice/crio-b603380b61ea1b6b9a9d411a9067e2dd41108242fa688a5928b0cab5507b7be4 WatchSource:0}: Error finding container b603380b61ea1b6b9a9d411a9067e2dd41108242fa688a5928b0cab5507b7be4: Status 404 returned error can't find the container with id b603380b61ea1b6b9a9d411a9067e2dd41108242fa688a5928b0cab5507b7be4
Jul 30 17:25:14.376321 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:14.376277    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerStarted","Data":"32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104"}
Jul 30 17:25:14.376503 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:14.376346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerStarted","Data":"b603380b61ea1b6b9a9d411a9067e2dd41108242fa688a5928b0cab5507b7be4"}
Jul 30 17:25:23.411307 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:23.411272    2569 generic.go:356] "Generic (PLEG): container finished" podID="36afb416-c271-4c66-bd80-495703396fb6" containerID="32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104" exitCode=0
Jul 30 17:25:23.411677 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:23.411318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerDied","Data":"32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104"}
Jul 30 17:25:24.417893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:24.417859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerStarted","Data":"86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a"}
Jul 30 17:25:24.418307 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:24.417923    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerStarted","Data":"05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15"}
Jul 30 17:25:24.418307 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:24.418106    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:24.438125 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:24.438074    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podStartSLOduration=2.625643032 podStartE2EDuration="11.438059018s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.812415986s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:25:24.437430213 +0000 UTC m=+1492.470240554" watchObservedRunningTime="2026-07-30 17:25:24.438059018 +0000 UTC m=+1492.470869356"
Jul 30 17:25:25.422269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:25.422223    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:25.423527 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:25.423498    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:25:26.427341 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:26.427306    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:25:31.431999 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:31.431961    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:31.432591 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:31.432567    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 30 17:25:41.433406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:41.433378    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:55.597218 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:55.597137    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"]
Jul 30 17:25:55.597709 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:55.597421    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container" containerID="cri-o://05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15" gracePeriod=30
Jul 30 17:25:55.597709 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:55.597480    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kube-rbac-proxy" containerID="cri-o://86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a" gracePeriod=30
Jul 30 17:25:56.427834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:56.427785    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 30 17:25:56.546167 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:56.546136    2569 generic.go:356] "Generic (PLEG): container finished" podID="36afb416-c271-4c66-bd80-495703396fb6" containerID="86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a" exitCode=2
Jul 30 17:25:56.546337 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:56.546214    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerDied","Data":"86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a"}
Jul 30 17:25:57.238389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.238360    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:57.343348 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343251    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-kube-rbac-proxy-sar-config\") pod \"36afb416-c271-4c66-bd80-495703396fb6\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") "
Jul 30 17:25:57.343348 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343308    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36afb416-c271-4c66-bd80-495703396fb6-proxy-tls\") pod \"36afb416-c271-4c66-bd80-495703396fb6\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") "
Jul 30 17:25:57.343525 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343431    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afb416-c271-4c66-bd80-495703396fb6-kserve-provision-location\") pod \"36afb416-c271-4c66-bd80-495703396fb6\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") "
Jul 30 17:25:57.343525 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343485    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-cabundle-cert\") pod \"36afb416-c271-4c66-bd80-495703396fb6\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") "
Jul 30 17:25:57.343525 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343514    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82ktx\" (UniqueName: \"kubernetes.io/projected/36afb416-c271-4c66-bd80-495703396fb6-kube-api-access-82ktx\") pod \"36afb416-c271-4c66-bd80-495703396fb6\" (UID: \"36afb416-c271-4c66-bd80-495703396fb6\") "
Jul 30 17:25:57.343774 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343745    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36afb416-c271-4c66-bd80-495703396fb6" (UID: "36afb416-c271-4c66-bd80-495703396fb6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:25:57.343915 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.343889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36afb416-c271-4c66-bd80-495703396fb6" (UID: "36afb416-c271-4c66-bd80-495703396fb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:25:57.344389 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.344367    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36afb416-c271-4c66-bd80-495703396fb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36afb416-c271-4c66-bd80-495703396fb6" (UID: "36afb416-c271-4c66-bd80-495703396fb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:25:57.345459 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.345435    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36afb416-c271-4c66-bd80-495703396fb6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36afb416-c271-4c66-bd80-495703396fb6" (UID: "36afb416-c271-4c66-bd80-495703396fb6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:25:57.345561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.345514    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36afb416-c271-4c66-bd80-495703396fb6-kube-api-access-82ktx" (OuterVolumeSpecName: "kube-api-access-82ktx") pod "36afb416-c271-4c66-bd80-495703396fb6" (UID: "36afb416-c271-4c66-bd80-495703396fb6"). InnerVolumeSpecName "kube-api-access-82ktx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:25:57.444757 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.444726    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:57.444757 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.444755    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36afb416-c271-4c66-bd80-495703396fb6-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:57.444935 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.444766    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36afb416-c271-4c66-bd80-495703396fb6-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:57.444935 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.444775    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36afb416-c271-4c66-bd80-495703396fb6-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:57.444935 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.444784    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82ktx\" (UniqueName: \"kubernetes.io/projected/36afb416-c271-4c66-bd80-495703396fb6-kube-api-access-82ktx\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:25:57.551554 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.551521    2569 generic.go:356] "Generic (PLEG): container finished" podID="36afb416-c271-4c66-bd80-495703396fb6" containerID="05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15" exitCode=0
Jul 30 17:25:57.551554 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.551557    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerDied","Data":"05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15"}
Jul 30 17:25:57.551744 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.551581    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"
Jul 30 17:25:57.551744 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.551603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb" event={"ID":"36afb416-c271-4c66-bd80-495703396fb6","Type":"ContainerDied","Data":"b603380b61ea1b6b9a9d411a9067e2dd41108242fa688a5928b0cab5507b7be4"}
Jul 30 17:25:57.551744 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.551629    2569 scope.go:117] "RemoveContainer" containerID="86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a"
Jul 30 17:25:57.560408 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.560381    2569 scope.go:117] "RemoveContainer" containerID="05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15"
Jul 30 17:25:57.567717 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.567697    2569 scope.go:117] "RemoveContainer" containerID="32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104"
Jul 30 17:25:57.574629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.574610    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"]
Jul 30 17:25:57.574890 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.574875    2569 scope.go:117] "RemoveContainer" containerID="86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a"
Jul 30 17:25:57.575154 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:25:57.575134    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a\": container with ID starting with 86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a not found: ID does not exist" containerID="86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a"
Jul 30 17:25:57.575208 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.575162    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a"} err="failed to get container status \"86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a\": rpc error: code = NotFound desc = could not find container \"86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a\": container with ID starting with 86bf8037035bde2b141e2173d55a48b18845c2a69402d8eaed9ad730f090e92a not found: ID does not exist"
Jul 30 17:25:57.575208 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.575180    2569 scope.go:117] "RemoveContainer" containerID="05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15"
Jul 30 17:25:57.575452 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:25:57.575431    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15\": container with ID starting with 05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15 not found: ID does not exist" containerID="05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15"
Jul 30 17:25:57.575556 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.575456    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15"} err="failed to get container status \"05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15\": rpc error: code = NotFound desc = could not find container \"05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15\": container with ID starting with 05decddfc6e3ea88b5e5c100cb7a938ff2258280fb7a965fdd81cc7991557b15 not found: ID does not exist"
Jul 30 17:25:57.575556 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.575474    2569 scope.go:117] "RemoveContainer" containerID="32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104"
Jul 30 17:25:57.575706 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:25:57.575686    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104\": container with ID starting with 32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104 not found: ID does not exist" containerID="32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104"
Jul 30 17:25:57.575763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.575717    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104"} err="failed to get container status \"32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104\": rpc error: code = NotFound desc = could not find container \"32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104\": container with ID starting with 32d3744c66415a159155da1aac65217bbf3576989d6bbb9876a278c3edf69104 not found: ID does not exist"
Jul 30 17:25:57.581049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:57.581026    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-j6lbb"]
Jul 30 17:25:58.652479 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:25:58.652442    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36afb416-c271-4c66-bd80-495703396fb6" path="/var/lib/kubelet/pods/36afb416-c271-4c66-bd80-495703396fb6/volumes"
Jul 30 17:26:00.759501 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759464    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"]
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759718    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="storage-initializer"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759730    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="storage-initializer"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759741    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kube-rbac-proxy"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759747    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kube-rbac-proxy"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759761    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759767    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759804    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kserve-container"
Jul 30 17:26:00.759962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.759813    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="36afb416-c271-4c66-bd80-495703396fb6" containerName="kube-rbac-proxy"
Jul 30 17:26:00.762634 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.762614    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.765036 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.765016    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:26:00.765238 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.765224    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:26:00.765320 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.765286    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:26:00.766057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.766042    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 30 17:26:00.766182 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.766166    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 30 17:26:00.766237 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.766188    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:26:00.766313 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.766235    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:26:00.778359 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.778339    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"]
Jul 30 17:26:00.870015 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.869979    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.870015 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.870020    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zg5t\" (UniqueName: \"kubernetes.io/projected/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-api-access-8zg5t\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.870228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.870044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.870228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.870072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/858d0e26-e881-4a15-8ecd-4d32614a76a7-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.870228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.870100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.970876 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.970834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.970925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.970969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zg5t\" (UniqueName: \"kubernetes.io/projected/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-api-access-8zg5t\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971069 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:26:00.970986    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 30 17:26:00.971069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.971003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.971029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/858d0e26-e881-4a15-8ecd-4d32614a76a7-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971069 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:26:00.971048    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls podName:858d0e26-e881-4a15-8ecd-4d32614a76a7 nodeName:}" failed. No retries permitted until 2026-07-30 17:26:01.471033243 +0000 UTC m=+1529.503843558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" (UID: "858d0e26-e881-4a15-8ecd-4d32614a76a7") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 30 17:26:00.971497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.971475    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/858d0e26-e881-4a15-8ecd-4d32614a76a7-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971675 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.971659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.971715 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.971673    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:00.982075 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:00.982048    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zg5t\" (UniqueName: \"kubernetes.io/projected/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-api-access-8zg5t\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:01.475206 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:01.475153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:01.477586 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:01.477564    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:01.673047 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:01.673006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:01.797606 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:01.797582    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"]
Jul 30 17:26:01.799442 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:26:01.799413    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858d0e26_e881_4a15_8ecd_4d32614a76a7.slice/crio-832f2a4808747ef2119d5a83f285d2f6492be04acc28df7263a676c9a85f2ca0 WatchSource:0}: Error finding container 832f2a4808747ef2119d5a83f285d2f6492be04acc28df7263a676c9a85f2ca0: Status 404 returned error can't find the container with id 832f2a4808747ef2119d5a83f285d2f6492be04acc28df7263a676c9a85f2ca0
Jul 30 17:26:02.574677 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:02.574639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerStarted","Data":"ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48"}
Jul 30 17:26:02.574889 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:02.574712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerStarted","Data":"832f2a4808747ef2119d5a83f285d2f6492be04acc28df7263a676c9a85f2ca0"}
Jul 30 17:26:06.589405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:06.589372    2569 generic.go:356] "Generic (PLEG): container finished" podID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerID="ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48" exitCode=0
Jul 30 17:26:06.589405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:06.589409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerDied","Data":"ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48"}
Jul 30 17:26:07.595698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:07.595662    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerStarted","Data":"61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0"}
Jul 30 17:26:07.596093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:07.595748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerStarted","Data":"26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd"}
Jul 30 17:26:07.596093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:07.596011    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:07.596189 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:07.596169    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:07.597502 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:07.597469    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:26:07.615973 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:07.615930    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" podStartSLOduration=2.9692895310000003 podStartE2EDuration="7.615917886s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.646628355s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:26:07.614702705 +0000 UTC m=+1535.647513041" watchObservedRunningTime="2026-07-30 17:26:07.615917886 +0000 UTC m=+1535.648728223"
Jul 30 17:26:08.599747 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:08.599704    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:26:13.604241 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:13.604213    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:13.604738 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:13.604708    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 30 17:26:23.606041 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:23.606012    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:35.960851 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:35.960818    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"]
Jul 30 17:26:35.961283 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:35.961052    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container" containerID="cri-o://26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd" gracePeriod=30
Jul 30 17:26:35.961283 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:35.961105    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kube-rbac-proxy" containerID="cri-o://61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0" gracePeriod=30
Jul 30 17:26:36.705206 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:36.705167    2569 generic.go:356] "Generic (PLEG): container finished" podID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerID="61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0" exitCode=2
Jul 30 17:26:36.705427 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:36.705232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerDied","Data":"61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0"}
Jul 30 17:26:37.207159 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.207135    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:37.374724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.374638    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-cabundle-cert\") pod \"858d0e26-e881-4a15-8ecd-4d32614a76a7\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") "
Jul 30 17:26:37.374724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.374675    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zg5t\" (UniqueName: \"kubernetes.io/projected/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-api-access-8zg5t\") pod \"858d0e26-e881-4a15-8ecd-4d32614a76a7\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") "
Jul 30 17:26:37.374724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.374696    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls\") pod \"858d0e26-e881-4a15-8ecd-4d32614a76a7\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") "
Jul 30 17:26:37.374942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.374742    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-rbac-proxy-sar-config\") pod \"858d0e26-e881-4a15-8ecd-4d32614a76a7\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") "
Jul 30 17:26:37.374942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.374806    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/858d0e26-e881-4a15-8ecd-4d32614a76a7-kserve-provision-location\") pod \"858d0e26-e881-4a15-8ecd-4d32614a76a7\" (UID: \"858d0e26-e881-4a15-8ecd-4d32614a76a7\") "
Jul 30 17:26:37.375166 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.375140    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "858d0e26-e881-4a15-8ecd-4d32614a76a7" (UID: "858d0e26-e881-4a15-8ecd-4d32614a76a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:26:37.375234 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.375169    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "858d0e26-e881-4a15-8ecd-4d32614a76a7" (UID: "858d0e26-e881-4a15-8ecd-4d32614a76a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:26:37.375909 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.375887    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858d0e26-e881-4a15-8ecd-4d32614a76a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "858d0e26-e881-4a15-8ecd-4d32614a76a7" (UID: "858d0e26-e881-4a15-8ecd-4d32614a76a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:26:37.376762 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.376741    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "858d0e26-e881-4a15-8ecd-4d32614a76a7" (UID: "858d0e26-e881-4a15-8ecd-4d32614a76a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:26:37.376852 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.376829    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-api-access-8zg5t" (OuterVolumeSpecName: "kube-api-access-8zg5t") pod "858d0e26-e881-4a15-8ecd-4d32614a76a7" (UID: "858d0e26-e881-4a15-8ecd-4d32614a76a7"). InnerVolumeSpecName "kube-api-access-8zg5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:26:37.475763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.475730    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/858d0e26-e881-4a15-8ecd-4d32614a76a7-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:26:37.475763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.475761    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:26:37.475946 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.475771    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zg5t\" (UniqueName: \"kubernetes.io/projected/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-api-access-8zg5t\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:26:37.475946 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.475781    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/858d0e26-e881-4a15-8ecd-4d32614a76a7-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:26:37.475946 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.475793    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/858d0e26-e881-4a15-8ecd-4d32614a76a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:26:37.711917 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.711879    2569 generic.go:356] "Generic (PLEG): container finished" podID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerID="26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd" exitCode=0
Jul 30 17:26:37.712103 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.711944    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"
Jul 30 17:26:37.712103 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.711953    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerDied","Data":"26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd"}
Jul 30 17:26:37.712103 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.712016    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl" event={"ID":"858d0e26-e881-4a15-8ecd-4d32614a76a7","Type":"ContainerDied","Data":"832f2a4808747ef2119d5a83f285d2f6492be04acc28df7263a676c9a85f2ca0"}
Jul 30 17:26:37.712103 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.712042    2569 scope.go:117] "RemoveContainer" containerID="61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0"
Jul 30 17:26:37.720519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.720497    2569 scope.go:117] "RemoveContainer" containerID="26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd"
Jul 30 17:26:37.727435 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.727418    2569 scope.go:117] "RemoveContainer" containerID="ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48"
Jul 30 17:26:37.733956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.733942    2569 scope.go:117] "RemoveContainer" containerID="61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0"
Jul 30 17:26:37.734209 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:26:37.734190    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0\": container with ID starting with 61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0 not found: ID does not exist" containerID="61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0"
Jul 30 17:26:37.734277 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.734218    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0"} err="failed to get container status \"61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0\": rpc error: code = NotFound desc = could not find container \"61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0\": container with ID starting with 61fe72b46d88f725a786de3720a1d2d34390aac48c0cd8797a661d56ceac41e0 not found: ID does not exist"
Jul 30 17:26:37.734277 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.734235    2569 scope.go:117] "RemoveContainer" containerID="26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd"
Jul 30 17:26:37.734487 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:26:37.734467    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd\": container with ID starting with 26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd not found: ID does not exist" containerID="26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd"
Jul 30 17:26:37.734558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.734497    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd"} err="failed to get container status \"26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd\": rpc error: code = NotFound desc = could not find container \"26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd\": container with ID starting with 26e0d06be904069ebc6e0745d68182e6debbb5cd873624595ead9273512cb0cd not found: ID does not exist"
Jul 30 17:26:37.734558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.734526    2569 scope.go:117] "RemoveContainer" containerID="ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48"
Jul 30 17:26:37.734744 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:26:37.734728    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48\": container with ID starting with ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48 not found: ID does not exist" containerID="ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48"
Jul 30 17:26:37.734786 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.734747    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48"} err="failed to get container status \"ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48\": rpc error: code = NotFound desc = could not find container \"ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48\": container with ID starting with ab9cc4b47ba7fc80b3f92f52f4c407875b70aa3c485eb5de01b566067016dc48 not found: ID does not exist"
Jul 30 17:26:37.739527 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.739505    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"]
Jul 30 17:26:37.744533 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:37.744514    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-44gkl"]
Jul 30 17:26:38.650937 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:38.650903    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" path="/var/lib/kubelet/pods/858d0e26-e881-4a15-8ecd-4d32614a76a7/volumes"
Jul 30 17:26:41.111825 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.111791    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"]
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112034    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="storage-initializer"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112045    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="storage-initializer"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112052    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112059    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112075    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kube-rbac-proxy"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112080    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kube-rbac-proxy"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112128    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kube-rbac-proxy"
Jul 30 17:26:41.112197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.112136    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="858d0e26-e881-4a15-8ecd-4d32614a76a7" containerName="kserve-container"
Jul 30 17:26:41.116762 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.116740    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.119468 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.119446    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:26:41.119468 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.119453    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 30 17:26:41.119617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.119468    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 30 17:26:41.119617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.119533    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:26:41.120409 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.120387    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:26:41.120518 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.120438    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:26:41.120518 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.120448    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:26:41.123201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.123178    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"]
Jul 30 17:26:41.126461 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.126441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.127594 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.127574    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"]
Jul 30 17:26:41.128714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.128695    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 30 17:26:41.136025 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.136003    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"]
Jul 30 17:26:41.301779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdvk\" (UniqueName: \"kubernetes.io/projected/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-api-access-csdvk\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301796    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301824    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d1c3-152b-46e0-ac46-9a7c66dd0825-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwflc\" (UniqueName: \"kubernetes.io/projected/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-api-access-bwflc\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301896    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kserve-provision-location\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.301965 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.302219 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.301999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-cabundle-cert\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.302219 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.302023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-proxy-tls\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kserve-provision-location\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403102    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-cabundle-cert\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-proxy-tls\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403157    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdvk\" (UniqueName: \"kubernetes.io/projected/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-api-access-csdvk\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d1c3-152b-46e0-ac46-9a7c66dd0825-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwflc\" (UniqueName: \"kubernetes.io/projected/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-api-access-bwflc\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403764 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403517    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kserve-provision-location\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403822 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403879 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.403941 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-cabundle-cert\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.403997 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.403970    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.404166 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.404142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.405841 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.405818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d1c3-152b-46e0-ac46-9a7c66dd0825-proxy-tls\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.405935 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.405917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-proxy-tls\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.413174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.413149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdvk\" (UniqueName: \"kubernetes.io/projected/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-api-access-csdvk\") pod \"isvc-canary-create-predictor-d9fd7b987-tb6rp\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.413282 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.413152    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwflc\" (UniqueName: \"kubernetes.io/projected/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-api-access-bwflc\") pod \"isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.428019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.428001    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:41.437145 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.437125    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:41.565066 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.565039    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"]
Jul 30 17:26:41.567577 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:26:41.567553    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0e2fd0_77fe_48b3_98f7_e8e157b2eb17.slice/crio-5dcfe027634d7ef84d3935b46e921da58c621344b08bbba640f522946ba32447 WatchSource:0}: Error finding container 5dcfe027634d7ef84d3935b46e921da58c621344b08bbba640f522946ba32447: Status 404 returned error can't find the container with id 5dcfe027634d7ef84d3935b46e921da58c621344b08bbba640f522946ba32447
Jul 30 17:26:41.582333 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.582281    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"]
Jul 30 17:26:41.584380 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:26:41.584354    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd968d1c3_152b_46e0_ac46_9a7c66dd0825.slice/crio-32d065956acf459b0c8beec785fee0d57a283dd80b5f7ef278e9bf73fb315e89 WatchSource:0}: Error finding container 32d065956acf459b0c8beec785fee0d57a283dd80b5f7ef278e9bf73fb315e89: Status 404 returned error can't find the container with id 32d065956acf459b0c8beec785fee0d57a283dd80b5f7ef278e9bf73fb315e89
Jul 30 17:26:41.729794 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.729750    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerStarted","Data":"3d95a51ab39ec2387c9dc42b2ff3b664153a9a16dde3763cfb471c9e0922fd3b"}
Jul 30 17:26:41.729978 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.729842    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerStarted","Data":"32d065956acf459b0c8beec785fee0d57a283dd80b5f7ef278e9bf73fb315e89"}
Jul 30 17:26:41.731517 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.731480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerStarted","Data":"b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa"}
Jul 30 17:26:41.731648 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:41.731541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerStarted","Data":"5dcfe027634d7ef84d3935b46e921da58c621344b08bbba640f522946ba32447"}
Jul 30 17:26:45.748639 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:45.748605    2569 generic.go:356] "Generic (PLEG): container finished" podID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerID="b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa" exitCode=0
Jul 30 17:26:45.749084 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:45.748678    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerDied","Data":"b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa"}
Jul 30 17:26:45.750364 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:45.750342    2569 generic.go:356] "Generic (PLEG): container finished" podID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerID="3d95a51ab39ec2387c9dc42b2ff3b664153a9a16dde3763cfb471c9e0922fd3b" exitCode=0
Jul 30 17:26:45.750453 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:45.750371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerDied","Data":"3d95a51ab39ec2387c9dc42b2ff3b664153a9a16dde3763cfb471c9e0922fd3b"}
Jul 30 17:26:46.755915 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.755869    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerStarted","Data":"3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799"}
Jul 30 17:26:46.756366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.755955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerStarted","Data":"2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd"}
Jul 30 17:26:46.756366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.756309    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:46.756516 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.756377    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:46.758028 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.757996    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:26:46.758148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.758007    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerStarted","Data":"eb348956e0a32e96a078d6aec3132916f441bc00421043214bf074f2d475a5f9"}
Jul 30 17:26:46.758148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.758136    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerStarted","Data":"8f866e4b7c2e43bbd365ad7559d1f5e11cb20c73020a6b7588dd884f12b11cad"}
Jul 30 17:26:46.758348 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.758332    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:46.775240 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.775196    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podStartSLOduration=2.014107489 podStartE2EDuration="5.775181413s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761073924s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:26:46.774147707 +0000 UTC m=+1574.806958045" watchObservedRunningTime="2026-07-30 17:26:46.775181413 +0000 UTC m=+1574.807991758"
Jul 30 17:26:46.793285 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:46.793225    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podStartSLOduration=2.048214704 podStartE2EDuration="5.793209047s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.744994343s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:26:46.790986062 +0000 UTC m=+1574.823796409" watchObservedRunningTime="2026-07-30 17:26:46.793209047 +0000 UTC m=+1574.826019457"
Jul 30 17:26:47.761847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:47.761810    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:47.761847 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:47.761820    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:26:47.762923 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:47.762900    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:26:48.766049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:48.766005    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:26:52.766792 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:52.766763    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:26:52.767400 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:52.767375    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:26:53.770494 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:53.770461    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:26:53.770975 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:26:53.770946    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:02.767440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:02.767398    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:03.771618 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:03.771576    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:12.767519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:12.767475    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:13.771297 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:13.771229    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:22.768117 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:22.768073    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:23.771820 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:23.771736    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:32.767581 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:32.767539    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:33.771440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:33.771399    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:42.768360 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:42.768319    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:43.771187 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:43.771143    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:52.767875 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:52.767829    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:53.771929 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:53.771885    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:27:55.647829 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:55.647787    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:27:57.647451 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:27:57.647406    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 30 17:28:05.649299 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:05.649243    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:28:07.648039 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:07.648008    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:28:11.211025 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:11.210987    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"]
Jul 30 17:28:11.211515 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:11.211265    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" containerID="cri-o://2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd" gracePeriod=30
Jul 30 17:28:11.211515 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:11.211304    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kube-rbac-proxy" containerID="cri-o://3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799" gracePeriod=30
Jul 30 17:28:11.222673 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:11.222648    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"]
Jul 30 17:28:11.222963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:11.222940    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container" containerID="cri-o://8f866e4b7c2e43bbd365ad7559d1f5e11cb20c73020a6b7588dd884f12b11cad" gracePeriod=30
Jul 30 17:28:11.222963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:11.222950    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kube-rbac-proxy" containerID="cri-o://eb348956e0a32e96a078d6aec3132916f441bc00421043214bf074f2d475a5f9" gracePeriod=30
Jul 30 17:28:12.087601 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:12.087570    2569 generic.go:356] "Generic (PLEG): container finished" podID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerID="eb348956e0a32e96a078d6aec3132916f441bc00421043214bf074f2d475a5f9" exitCode=2
Jul 30 17:28:12.087771 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:12.087638    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerDied","Data":"eb348956e0a32e96a078d6aec3132916f441bc00421043214bf074f2d475a5f9"}
Jul 30 17:28:12.089652 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:12.089630    2569 generic.go:356] "Generic (PLEG): container finished" podID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerID="3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799" exitCode=2
Jul 30 17:28:12.089762 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:12.089703    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerDied","Data":"3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799"}
Jul 30 17:28:12.761862 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:12.761823    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 30 17:28:13.766497 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:13.766445    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 30 17:28:15.648268 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:15.648220    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 30 17:28:16.106338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.106308    2569 generic.go:356] "Generic (PLEG): container finished" podID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerID="8f866e4b7c2e43bbd365ad7559d1f5e11cb20c73020a6b7588dd884f12b11cad" exitCode=0
Jul 30 17:28:16.106469 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.106385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerDied","Data":"8f866e4b7c2e43bbd365ad7559d1f5e11cb20c73020a6b7588dd884f12b11cad"}
Jul 30 17:28:16.258509 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.258484    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:28:16.261107 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.261088    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:28:16.327792 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327715    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-cabundle-cert\") pod \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") "
Jul 30 17:28:16.327792 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327749    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kserve-provision-location\") pod \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") "
Jul 30 17:28:16.327792 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327772    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-proxy-tls\") pod \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327798    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-rbac-proxy-sar-config\") pod \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327832    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-rbac-proxy-sar-config\") pod \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327879    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-cabundle-cert\") pod \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327912    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kserve-provision-location\") pod \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327967    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csdvk\" (UniqueName: \"kubernetes.io/projected/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-api-access-csdvk\") pod \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\" (UID: \"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.327993    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d1c3-152b-46e0-ac46-9a7c66dd0825-proxy-tls\") pod \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328024    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwflc\" (UniqueName: \"kubernetes.io/projected/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-api-access-bwflc\") pod \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\" (UID: \"d968d1c3-152b-46e0-ac46-9a7c66dd0825\") "
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328087    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d968d1c3-152b-46e0-ac46-9a7c66dd0825" (UID: "d968d1c3-152b-46e0-ac46-9a7c66dd0825"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328180    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" (UID: "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328397    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.328406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328405    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" (UID: "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:28:16.328885 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328420    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.328885 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328514    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d968d1c3-152b-46e0-ac46-9a7c66dd0825" (UID: "d968d1c3-152b-46e0-ac46-9a7c66dd0825"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:28:16.328885 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.328804    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" (UID: "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:28:16.329227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.329204    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d968d1c3-152b-46e0-ac46-9a7c66dd0825" (UID: "d968d1c3-152b-46e0-ac46-9a7c66dd0825"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:28:16.330608 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.330586    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" (UID: "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:28:16.330717 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.330604    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-api-access-csdvk" (OuterVolumeSpecName: "kube-api-access-csdvk") pod "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" (UID: "dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17"). InnerVolumeSpecName "kube-api-access-csdvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:28:16.330717 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.330605    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d968d1c3-152b-46e0-ac46-9a7c66dd0825-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d968d1c3-152b-46e0-ac46-9a7c66dd0825" (UID: "d968d1c3-152b-46e0-ac46-9a7c66dd0825"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:28:16.331057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.331041    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-api-access-bwflc" (OuterVolumeSpecName: "kube-api-access-bwflc") pod "d968d1c3-152b-46e0-ac46-9a7c66dd0825" (UID: "d968d1c3-152b-46e0-ac46-9a7c66dd0825"). InnerVolumeSpecName "kube-api-access-bwflc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:28:16.403203 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403175    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"]
Jul 30 17:28:16.403538 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403519    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="storage-initializer"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403542    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="storage-initializer"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403559    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kube-rbac-proxy"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403568    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kube-rbac-proxy"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403586    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kube-rbac-proxy"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403596    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kube-rbac-proxy"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403610    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="storage-initializer"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403618    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="storage-initializer"
Jul 30 17:28:16.403629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403626    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403635    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403652    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403661    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403728    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kube-rbac-proxy"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403739    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kserve-container"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403748    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerName="kube-rbac-proxy"
Jul 30 17:28:16.404019 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.403759    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" containerName="kserve-container"
Jul 30 17:28:16.408148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.408129    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"]
Jul 30 17:28:16.408276 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.408233    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.410671 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.410646    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 30 17:28:16.410769 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.410672    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 30 17:28:16.411616 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.411602    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.413763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.413745    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 30 17:28:16.428742 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428724    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428742 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428743    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428753    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdvk\" (UniqueName: \"kubernetes.io/projected/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-kube-api-access-csdvk\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428762    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d968d1c3-152b-46e0-ac46-9a7c66dd0825-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428772    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwflc\" (UniqueName: \"kubernetes.io/projected/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-api-access-bwflc\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428780    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428788    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.428857 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.428796    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d968d1c3-152b-46e0-ac46-9a7c66dd0825-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:28:16.437475 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.437455    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"]
Jul 30 17:28:16.439199 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.439180    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"]
Jul 30 17:28:16.529883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.529852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9873b168-4453-4fc1-adbf-dc79453a79d5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.529883 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.529883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/308c8752-b50e-40cc-9d19-ab53792c84a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.530074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.529907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgzg\" (UniqueName: \"kubernetes.io/projected/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-api-access-ttgzg\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.530074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.529999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-cabundle-cert\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.530074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.530018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9873b168-4453-4fc1-adbf-dc79453a79d5-proxy-tls\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.530074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.530039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.530202 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.530074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr54\" (UniqueName: \"kubernetes.io/projected/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-api-access-bvr54\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.530202 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.530097    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.530202 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.530150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/308c8752-b50e-40cc-9d19-ab53792c84a0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.530202 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.530181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631142 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/308c8752-b50e-40cc-9d19-ab53792c84a0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631142 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631142 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9873b168-4453-4fc1-adbf-dc79453a79d5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.631142 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/308c8752-b50e-40cc-9d19-ab53792c84a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631172    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgzg\" (UniqueName: \"kubernetes.io/projected/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-api-access-ttgzg\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-cabundle-cert\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.631446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631279    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9873b168-4453-4fc1-adbf-dc79453a79d5-proxy-tls\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.631446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631306    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr54\" (UniqueName: \"kubernetes.io/projected/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-api-access-bvr54\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.631446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631369    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.631763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/308c8752-b50e-40cc-9d19-ab53792c84a0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.631763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631593    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9873b168-4453-4fc1-adbf-dc79453a79d5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.631955 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631926    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.632055 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.631988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-cabundle-cert\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.632055 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.632013    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.632201 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.632185    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.633880 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.633858    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/308c8752-b50e-40cc-9d19-ab53792c84a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.633951 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.633898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9873b168-4453-4fc1-adbf-dc79453a79d5-proxy-tls\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.643117 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.643088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr54\" (UniqueName: \"kubernetes.io/projected/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-api-access-bvr54\") pod \"isvc-canary-promote-predictor-85f6464bd7-9gh22\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.643391 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.643374    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgzg\" (UniqueName: \"kubernetes.io/projected/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-api-access-ttgzg\") pod \"isvc-canary-promote-v2-predictor-66f6768f84-5vxlq\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.719303 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.719246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:16.724599 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.724581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:16.859736 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.859686    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"]
Jul 30 17:28:16.862353 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:28:16.862326    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9873b168_4453_4fc1_adbf_dc79453a79d5.slice/crio-e06ba4472f66b194ed16037788dadc464b4ff15da00489e9a3dcecab15f431d6 WatchSource:0}: Error finding container e06ba4472f66b194ed16037788dadc464b4ff15da00489e9a3dcecab15f431d6: Status 404 returned error can't find the container with id e06ba4472f66b194ed16037788dadc464b4ff15da00489e9a3dcecab15f431d6
Jul 30 17:28:16.880025 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:16.879954    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"]
Jul 30 17:28:16.882370 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:28:16.882314    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308c8752_b50e_40cc_9d19_ab53792c84a0.slice/crio-0721acab706d8d74c8718e3abb3432fe74d3ccf994b165b2711a87597c085abd WatchSource:0}: Error finding container 0721acab706d8d74c8718e3abb3432fe74d3ccf994b165b2711a87597c085abd: Status 404 returned error can't find the container with id 0721acab706d8d74c8718e3abb3432fe74d3ccf994b165b2711a87597c085abd
Jul 30 17:28:17.120151 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.120111    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx" event={"ID":"d968d1c3-152b-46e0-ac46-9a7c66dd0825","Type":"ContainerDied","Data":"32d065956acf459b0c8beec785fee0d57a283dd80b5f7ef278e9bf73fb315e89"}
Jul 30 17:28:17.120338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.120127    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"
Jul 30 17:28:17.120338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.120238    2569 scope.go:117] "RemoveContainer" containerID="eb348956e0a32e96a078d6aec3132916f441bc00421043214bf074f2d475a5f9"
Jul 30 17:28:17.122111 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.122081    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerStarted","Data":"13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf"}
Jul 30 17:28:17.122243 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.122160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerStarted","Data":"0721acab706d8d74c8718e3abb3432fe74d3ccf994b165b2711a87597c085abd"}
Jul 30 17:28:17.124980 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.124943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerStarted","Data":"70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4"}
Jul 30 17:28:17.125107 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.124995    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerStarted","Data":"e06ba4472f66b194ed16037788dadc464b4ff15da00489e9a3dcecab15f431d6"}
Jul 30 17:28:17.127336 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.127308    2569 generic.go:356] "Generic (PLEG): container finished" podID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" containerID="2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd" exitCode=0
Jul 30 17:28:17.127445 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.127347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerDied","Data":"2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd"}
Jul 30 17:28:17.127445 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.127369    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"
Jul 30 17:28:17.127445 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.127396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp" event={"ID":"dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17","Type":"ContainerDied","Data":"5dcfe027634d7ef84d3935b46e921da58c621344b08bbba640f522946ba32447"}
Jul 30 17:28:17.130475 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.130457    2569 scope.go:117] "RemoveContainer" containerID="8f866e4b7c2e43bbd365ad7559d1f5e11cb20c73020a6b7588dd884f12b11cad"
Jul 30 17:28:17.139493 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.139454    2569 scope.go:117] "RemoveContainer" containerID="3d95a51ab39ec2387c9dc42b2ff3b664153a9a16dde3763cfb471c9e0922fd3b"
Jul 30 17:28:17.142960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.142935    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"]
Jul 30 17:28:17.147219 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.147196    2569 scope.go:117] "RemoveContainer" containerID="3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799"
Jul 30 17:28:17.151940 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.151915    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-59d4d8d7b6-8jfhx"]
Jul 30 17:28:17.156402 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.156374    2569 scope.go:117] "RemoveContainer" containerID="2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd"
Jul 30 17:28:17.164772 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.164753    2569 scope.go:117] "RemoveContainer" containerID="b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa"
Jul 30 17:28:17.174385 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.174314    2569 scope.go:117] "RemoveContainer" containerID="3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799"
Jul 30 17:28:17.174644 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:28:17.174624    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799\": container with ID starting with 3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799 not found: ID does not exist" containerID="3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799"
Jul 30 17:28:17.174721 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.174658    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799"} err="failed to get container status \"3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799\": rpc error: code = NotFound desc = could not find container \"3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799\": container with ID starting with 3538ea5917276c8b069d2b74e256c769dc7fe7d3083d16af5a1773eb05e12799 not found: ID does not exist"
Jul 30 17:28:17.174721 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.174677    2569 scope.go:117] "RemoveContainer" containerID="2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd"
Jul 30 17:28:17.174936 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:28:17.174916    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd\": container with ID starting with 2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd not found: ID does not exist" containerID="2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd"
Jul 30 17:28:17.175071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.174937    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd"} err="failed to get container status \"2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd\": rpc error: code = NotFound desc = could not find container \"2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd\": container with ID starting with 2ed4c26d28bc63f031af9e350e9c07f30e536c54ee8c95e628c26048705eedbd not found: ID does not exist"
Jul 30 17:28:17.175071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.174951    2569 scope.go:117] "RemoveContainer" containerID="b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa"
Jul 30 17:28:17.175249 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:28:17.175228    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa\": container with ID starting with b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa not found: ID does not exist" containerID="b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa"
Jul 30 17:28:17.175367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.175266    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa"} err="failed to get container status \"b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa\": rpc error: code = NotFound desc = could not find container \"b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa\": container with ID starting with b375c6860be4594f1d7cc7c8b6e8fbc8a31626a12e92efd388d2477ed9317eaa not found: ID does not exist"
Jul 30 17:28:17.215285 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.214597    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"]
Jul 30 17:28:17.216589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:17.216560    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-d9fd7b987-tb6rp"]
Jul 30 17:28:18.652137 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:18.652108    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d968d1c3-152b-46e0-ac46-9a7c66dd0825" path="/var/lib/kubelet/pods/d968d1c3-152b-46e0-ac46-9a7c66dd0825/volumes"
Jul 30 17:28:18.652616 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:18.652600    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17" path="/var/lib/kubelet/pods/dd0e2fd0-77fe-48b3-98f7-e8e157b2eb17/volumes"
Jul 30 17:28:21.145141 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:21.145106    2569 generic.go:356] "Generic (PLEG): container finished" podID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerID="13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf" exitCode=0
Jul 30 17:28:21.145598 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:21.145181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerDied","Data":"13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf"}
Jul 30 17:28:21.146874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:21.146853    2569 generic.go:356] "Generic (PLEG): container finished" podID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerID="70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4" exitCode=0
Jul 30 17:28:21.146967 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:21.146924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerDied","Data":"70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4"}
Jul 30 17:28:22.152193 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.152155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerStarted","Data":"5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63"}
Jul 30 17:28:22.152664 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.152223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerStarted","Data":"3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338"}
Jul 30 17:28:22.152664 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.152555    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:22.152788 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.152726    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:22.154242 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.154214    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:28:22.154548 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.154526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerStarted","Data":"563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6"}
Jul 30 17:28:22.154601 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.154576    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerStarted","Data":"2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048"}
Jul 30 17:28:22.154766 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.154750    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:22.154834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.154802    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:22.155860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.155834    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:28:22.175808 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.175765    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podStartSLOduration=2.516511676 podStartE2EDuration="6.175752396s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.65924072s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:28:22.174152841 +0000 UTC m=+1670.206963204" watchObservedRunningTime="2026-07-30 17:28:22.175752396 +0000 UTC m=+1670.208562734"
Jul 30 17:28:22.197353 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:22.197307    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podStartSLOduration=2.454509817 podStartE2EDuration="6.197296235s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742786418s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:28:22.193109165 +0000 UTC m=+1670.225919513" watchObservedRunningTime="2026-07-30 17:28:22.197296235 +0000 UTC m=+1670.230106574"
Jul 30 17:28:23.159031 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:23.158989    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:28:23.159455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:23.159100    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:28:28.163427 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:28.163392    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:28:28.164692 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:28.164341    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:28:28.164828 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:28.164810    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:28:28.165319 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:28.165293    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:28:38.164664 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:38.164622    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:28:38.165875 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:38.165850    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:28:48.164967 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:48.164923    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:28:48.165430 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:48.165245    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:28:58.165162 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:58.165080    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:28:58.165541 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:28:58.165270    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:29:08.164399 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:08.164363    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:29:08.165582 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:08.165554    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:29:18.164960 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:18.164915    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:29:18.165455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:18.165207    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:29:28.165016 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:28.164979    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:29:28.165484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:28.165210    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:29:31.647735 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:31.647693    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 30 17:29:38.166301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:38.166244    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:29:41.648876 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:41.648845    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:29:46.475520 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:46.475488    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"]
Jul 30 17:29:46.475897 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:46.475712    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container" containerID="cri-o://2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048" gracePeriod=30
Jul 30 17:29:46.475897 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:46.475776    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kube-rbac-proxy" containerID="cri-o://563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6" gracePeriod=30
Jul 30 17:29:47.480942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:47.480911    2569 generic.go:356] "Generic (PLEG): container finished" podID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerID="563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6" exitCode=2
Jul 30 17:29:47.481353 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:47.480991    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerDied","Data":"563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6"}
Jul 30 17:29:48.159908 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:48.159863    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 30 17:29:51.113867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.113842    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:29:51.207509 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207475    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-cabundle-cert\") pod \"9873b168-4453-4fc1-adbf-dc79453a79d5\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") "
Jul 30 17:29:51.207686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207535    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9873b168-4453-4fc1-adbf-dc79453a79d5-proxy-tls\") pod \"9873b168-4453-4fc1-adbf-dc79453a79d5\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") "
Jul 30 17:29:51.207686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207582    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9873b168-4453-4fc1-adbf-dc79453a79d5-kserve-provision-location\") pod \"9873b168-4453-4fc1-adbf-dc79453a79d5\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") "
Jul 30 17:29:51.207686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207608    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvr54\" (UniqueName: \"kubernetes.io/projected/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-api-access-bvr54\") pod \"9873b168-4453-4fc1-adbf-dc79453a79d5\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") "
Jul 30 17:29:51.207859 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207736    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-rbac-proxy-sar-config\") pod \"9873b168-4453-4fc1-adbf-dc79453a79d5\" (UID: \"9873b168-4453-4fc1-adbf-dc79453a79d5\") "
Jul 30 17:29:51.207933 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207865    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9873b168-4453-4fc1-adbf-dc79453a79d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9873b168-4453-4fc1-adbf-dc79453a79d5" (UID: "9873b168-4453-4fc1-adbf-dc79453a79d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:29:51.207933 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.207885    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9873b168-4453-4fc1-adbf-dc79453a79d5" (UID: "9873b168-4453-4fc1-adbf-dc79453a79d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:29:51.208079 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.208058    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:29:51.208079 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.208077    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9873b168-4453-4fc1-adbf-dc79453a79d5-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:29:51.208295 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.208070    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9873b168-4453-4fc1-adbf-dc79453a79d5" (UID: "9873b168-4453-4fc1-adbf-dc79453a79d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:29:51.209712 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.209686    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9873b168-4453-4fc1-adbf-dc79453a79d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9873b168-4453-4fc1-adbf-dc79453a79d5" (UID: "9873b168-4453-4fc1-adbf-dc79453a79d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:29:51.209796 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.209762    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-api-access-bvr54" (OuterVolumeSpecName: "kube-api-access-bvr54") pod "9873b168-4453-4fc1-adbf-dc79453a79d5" (UID: "9873b168-4453-4fc1-adbf-dc79453a79d5"). InnerVolumeSpecName "kube-api-access-bvr54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:29:51.308952 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.308912    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9873b168-4453-4fc1-adbf-dc79453a79d5-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:29:51.308952 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.308942    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvr54\" (UniqueName: \"kubernetes.io/projected/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-api-access-bvr54\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:29:51.308952 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.308952    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9873b168-4453-4fc1-adbf-dc79453a79d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:29:51.497274 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.497160    2569 generic.go:356] "Generic (PLEG): container finished" podID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerID="2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048" exitCode=0
Jul 30 17:29:51.497274 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.497209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerDied","Data":"2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048"}
Jul 30 17:29:51.497274 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.497225    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"
Jul 30 17:29:51.497508 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.497289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22" event={"ID":"9873b168-4453-4fc1-adbf-dc79453a79d5","Type":"ContainerDied","Data":"e06ba4472f66b194ed16037788dadc464b4ff15da00489e9a3dcecab15f431d6"}
Jul 30 17:29:51.497508 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.497333    2569 scope.go:117] "RemoveContainer" containerID="563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6"
Jul 30 17:29:51.505410 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.505393    2569 scope.go:117] "RemoveContainer" containerID="2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048"
Jul 30 17:29:51.513754 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.513664    2569 scope.go:117] "RemoveContainer" containerID="70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4"
Jul 30 17:29:51.521040 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.521017    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"]
Jul 30 17:29:51.521686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.521673    2569 scope.go:117] "RemoveContainer" containerID="563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6"
Jul 30 17:29:51.521943 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:29:51.521919    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6\": container with ID starting with 563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6 not found: ID does not exist" containerID="563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6"
Jul 30 17:29:51.522031 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.521950    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6"} err="failed to get container status \"563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6\": rpc error: code = NotFound desc = could not find container \"563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6\": container with ID starting with 563463a5e41201d1a0e06afb01ba0ca522c065861eebbbd0b80753890a9641c6 not found: ID does not exist"
Jul 30 17:29:51.522031 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.521970    2569 scope.go:117] "RemoveContainer" containerID="2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048"
Jul 30 17:29:51.522214 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:29:51.522197    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048\": container with ID starting with 2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048 not found: ID does not exist" containerID="2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048"
Jul 30 17:29:51.522278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.522220    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048"} err="failed to get container status \"2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048\": rpc error: code = NotFound desc = could not find container \"2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048\": container with ID starting with 2f4d761d7366986069644f4766a78aee9d943d1c2d771fb2e583f049f9c12048 not found: ID does not exist"
Jul 30 17:29:51.522278 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.522235    2569 scope.go:117] "RemoveContainer" containerID="70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4"
Jul 30 17:29:51.522481 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:29:51.522466    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4\": container with ID starting with 70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4 not found: ID does not exist" containerID="70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4"
Jul 30 17:29:51.522561 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.522484    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4"} err="failed to get container status \"70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4\": rpc error: code = NotFound desc = could not find container \"70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4\": container with ID starting with 70d79e2755b6f5da3aff3be9c4ca5b1ed79424cbcd77b3661e1b48868162b3f4 not found: ID does not exist"
Jul 30 17:29:51.525350 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:51.525329    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-85f6464bd7-9gh22"]
Jul 30 17:29:52.652047 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:29:52.652017    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" path="/var/lib/kubelet/pods/9873b168-4453-4fc1-adbf-dc79453a79d5/volumes"
Jul 30 17:31:26.854239 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:26.854205    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"]
Jul 30 17:31:26.854843 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:26.854469    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" containerID="cri-o://3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338" gracePeriod=30
Jul 30 17:31:26.854843 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:26.854506    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kube-rbac-proxy" containerID="cri-o://5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63" gracePeriod=30
Jul 30 17:31:27.852670 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:27.852636    2569 generic.go:356] "Generic (PLEG): container finished" podID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerID="5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63" exitCode=2
Jul 30 17:31:27.852849 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:27.852677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerDied","Data":"5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63"}
Jul 30 17:31:28.159680 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:28.159595    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 30 17:31:28.166047 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:28.166015    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 30 17:31:31.489752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.489724    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:31:31.581123 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581051    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-cabundle-cert\") pod \"308c8752-b50e-40cc-9d19-ab53792c84a0\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") "
Jul 30 17:31:31.581123 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581093    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttgzg\" (UniqueName: \"kubernetes.io/projected/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-api-access-ttgzg\") pod \"308c8752-b50e-40cc-9d19-ab53792c84a0\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") "
Jul 30 17:31:31.581367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581208    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-rbac-proxy-sar-config\") pod \"308c8752-b50e-40cc-9d19-ab53792c84a0\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") "
Jul 30 17:31:31.581367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581275    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/308c8752-b50e-40cc-9d19-ab53792c84a0-kserve-provision-location\") pod \"308c8752-b50e-40cc-9d19-ab53792c84a0\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") "
Jul 30 17:31:31.581367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/308c8752-b50e-40cc-9d19-ab53792c84a0-proxy-tls\") pod \"308c8752-b50e-40cc-9d19-ab53792c84a0\" (UID: \"308c8752-b50e-40cc-9d19-ab53792c84a0\") "
Jul 30 17:31:31.581547 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581523    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "308c8752-b50e-40cc-9d19-ab53792c84a0" (UID: "308c8752-b50e-40cc-9d19-ab53792c84a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:31:31.581601 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581545    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "308c8752-b50e-40cc-9d19-ab53792c84a0" (UID: "308c8752-b50e-40cc-9d19-ab53792c84a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:31:31.581642 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.581611    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/308c8752-b50e-40cc-9d19-ab53792c84a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "308c8752-b50e-40cc-9d19-ab53792c84a0" (UID: "308c8752-b50e-40cc-9d19-ab53792c84a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:31:31.583227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.583203    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308c8752-b50e-40cc-9d19-ab53792c84a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "308c8752-b50e-40cc-9d19-ab53792c84a0" (UID: "308c8752-b50e-40cc-9d19-ab53792c84a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:31:31.583311 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.583211    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-api-access-ttgzg" (OuterVolumeSpecName: "kube-api-access-ttgzg") pod "308c8752-b50e-40cc-9d19-ab53792c84a0" (UID: "308c8752-b50e-40cc-9d19-ab53792c84a0"). InnerVolumeSpecName "kube-api-access-ttgzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:31:31.682672 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.682635    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.682672 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.682667    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgzg\" (UniqueName: \"kubernetes.io/projected/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-api-access-ttgzg\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.682874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.682683    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/308c8752-b50e-40cc-9d19-ab53792c84a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.682874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.682695    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/308c8752-b50e-40cc-9d19-ab53792c84a0-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.682874 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.682707    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/308c8752-b50e-40cc-9d19-ab53792c84a0-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:31:31.870026 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.869936    2569 generic.go:356] "Generic (PLEG): container finished" podID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerID="3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338" exitCode=0
Jul 30 17:31:31.870026 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.869989    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"
Jul 30 17:31:31.870220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.869988    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerDied","Data":"3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338"}
Jul 30 17:31:31.870220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.870149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq" event={"ID":"308c8752-b50e-40cc-9d19-ab53792c84a0","Type":"ContainerDied","Data":"0721acab706d8d74c8718e3abb3432fe74d3ccf994b165b2711a87597c085abd"}
Jul 30 17:31:31.870220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.870200    2569 scope.go:117] "RemoveContainer" containerID="5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63"
Jul 30 17:31:31.879623 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.879605    2569 scope.go:117] "RemoveContainer" containerID="3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338"
Jul 30 17:31:31.888584 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.888561    2569 scope.go:117] "RemoveContainer" containerID="13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf"
Jul 30 17:31:31.895707 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.895688    2569 scope.go:117] "RemoveContainer" containerID="5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63"
Jul 30 17:31:31.895975 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:31:31.895955    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63\": container with ID starting with 5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63 not found: ID does not exist" containerID="5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63"
Jul 30 17:31:31.896027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.895987    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63"} err="failed to get container status \"5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63\": rpc error: code = NotFound desc = could not find container \"5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63\": container with ID starting with 5dde6a1b0beb7067e4aa0bb58ef4c2612ffef4e66471003d477879b713d57a63 not found: ID does not exist"
Jul 30 17:31:31.896027 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.896007    2569 scope.go:117] "RemoveContainer" containerID="3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338"
Jul 30 17:31:31.896250 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:31:31.896230    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338\": container with ID starting with 3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338 not found: ID does not exist" containerID="3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338"
Jul 30 17:31:31.896250 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.896272    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338"} err="failed to get container status \"3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338\": rpc error: code = NotFound desc = could not find container \"3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338\": container with ID starting with 3638f6cad30fcdf530347a461ac51b6840726d7c442f7beb1cbfe931527ec338 not found: ID does not exist"
Jul 30 17:31:31.896250 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.896293    2569 scope.go:117] "RemoveContainer" containerID="13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf"
Jul 30 17:31:31.896567 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:31:31.896550    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf\": container with ID starting with 13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf not found: ID does not exist" containerID="13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf"
Jul 30 17:31:31.896618 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.896574    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf"} err="failed to get container status \"13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf\": rpc error: code = NotFound desc = could not find container \"13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf\": container with ID starting with 13e4b6493d427ee64be86cfe845c2709565eac8edee8923e42fd27af76a196cf not found: ID does not exist"
Jul 30 17:31:31.907152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.907128    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"]
Jul 30 17:31:31.912655 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:31.912630    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-66f6768f84-5vxlq"]
Jul 30 17:31:32.655240 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:31:32.655211    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" path="/var/lib/kubelet/pods/308c8752-b50e-40cc-9d19-ab53792c84a0/volumes"
Jul 30 17:33:01.981499 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981461    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"]
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981733    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="storage-initializer"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981744    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="storage-initializer"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981756    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="storage-initializer"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981761    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="storage-initializer"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981768    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kube-rbac-proxy"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981773    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kube-rbac-proxy"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981780    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kube-rbac-proxy"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981785    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kube-rbac-proxy"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981794    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981798    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981808    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981814    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981852    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kube-rbac-proxy"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981860    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kserve-container"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981866    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9873b168-4453-4fc1-adbf-dc79453a79d5" containerName="kserve-container"
Jul 30 17:33:01.981983 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.981873    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="308c8752-b50e-40cc-9d19-ab53792c84a0" containerName="kube-rbac-proxy"
Jul 30 17:33:01.984925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.984903    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:01.987519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.987497    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:33:01.988485 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.988463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:33:01.988580 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.988465    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 30 17:33:01.988640 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.988572    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:33:01.988640 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.988586    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:33:01.988640 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.988597    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 30 17:33:01.988774 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.988684    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:33:01.994657 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:01.994631    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"]
Jul 30 17:33:02.089448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.089416    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a82406-947b-4699-afc0-1794f90d33e7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.089448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.089447    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258wv\" (UniqueName: \"kubernetes.io/projected/a8a82406-947b-4699-afc0-1794f90d33e7-kube-api-access-258wv\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.089639 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.089473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.089639 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.089565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a82406-947b-4699-afc0-1794f90d33e7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.089639 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.089594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.190779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.190745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258wv\" (UniqueName: \"kubernetes.io/projected/a8a82406-947b-4699-afc0-1794f90d33e7-kube-api-access-258wv\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.190912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.190787    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.190912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.190836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a82406-947b-4699-afc0-1794f90d33e7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.190912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.190864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.190912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.190892    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a82406-947b-4699-afc0-1794f90d33e7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.191325 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.191300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a82406-947b-4699-afc0-1794f90d33e7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.191590 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.191567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.191590 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.191570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.193412 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.193386    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a82406-947b-4699-afc0-1794f90d33e7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.200437 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.200417    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258wv\" (UniqueName: \"kubernetes.io/projected/a8a82406-947b-4699-afc0-1794f90d33e7-kube-api-access-258wv\") pod \"isvc-canary-rollback-predictor-7c9c87bcff-49q58\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.295637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.295546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:02.422607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:02.422574    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"]
Jul 30 17:33:02.425627 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:33:02.425598    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a82406_947b_4699_afc0_1794f90d33e7.slice/crio-f001eb320af1e0e1384bf24ad341854a2b5e27c283a25a329aae6f9d29109ef7 WatchSource:0}: Error finding container f001eb320af1e0e1384bf24ad341854a2b5e27c283a25a329aae6f9d29109ef7: Status 404 returned error can't find the container with id f001eb320af1e0e1384bf24ad341854a2b5e27c283a25a329aae6f9d29109ef7
Jul 30 17:33:03.204469 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:03.204438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerStarted","Data":"d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7"}
Jul 30 17:33:03.204848 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:03.204500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerStarted","Data":"f001eb320af1e0e1384bf24ad341854a2b5e27c283a25a329aae6f9d29109ef7"}
Jul 30 17:33:07.220341 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:07.220299    2569 generic.go:356] "Generic (PLEG): container finished" podID="a8a82406-947b-4699-afc0-1794f90d33e7" containerID="d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7" exitCode=0
Jul 30 17:33:07.220709 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:07.220373    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerDied","Data":"d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7"}
Jul 30 17:33:08.225829 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:08.225795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerStarted","Data":"c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c"}
Jul 30 17:33:08.226294 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:08.225856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerStarted","Data":"1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba"}
Jul 30 17:33:08.226294 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:08.226191    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:08.226417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:08.226321    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:08.227544 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:08.227523    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:08.246511 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:08.246465    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podStartSLOduration=3.472612564 podStartE2EDuration="7.246452184s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77383962s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:33:08.244942797 +0000 UTC m=+1956.277753134" watchObservedRunningTime="2026-07-30 17:33:08.246452184 +0000 UTC m=+1956.279262524"
Jul 30 17:33:09.229584 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:09.229547    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:10.233373 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:10.233328    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:15.238328 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:15.238289    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:33:15.238860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:15.238833    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:25.239810 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:25.239717    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:35.239231 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:35.239191    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:45.238979 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:45.238937    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:33:55.238932 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:33:55.238890    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:34:05.239194 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:05.239155    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:34:15.239406 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:15.239368    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:34:25.240334 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:25.240297    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:34:32.172615 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:32.172581    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"]
Jul 30 17:34:32.173177 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:32.172900    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kube-rbac-proxy" containerID="cri-o://c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c" gracePeriod=30
Jul 30 17:34:32.173177 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:32.172901    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" containerID="cri-o://1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba" gracePeriod=30
Jul 30 17:34:32.528944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:32.528912    2569 generic.go:356] "Generic (PLEG): container finished" podID="a8a82406-947b-4699-afc0-1794f90d33e7" containerID="c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c" exitCode=2
Jul 30 17:34:32.529130 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:32.528977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerDied","Data":"c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c"}
Jul 30 17:34:35.234351 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:35.234309    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 30 17:34:35.239752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:35.239723    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 30 17:34:36.917902 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:36.917877    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:34:37.108137 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108037    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a82406-947b-4699-afc0-1794f90d33e7-kserve-provision-location\") pod \"a8a82406-947b-4699-afc0-1794f90d33e7\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") "
Jul 30 17:34:37.108137 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108095    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-258wv\" (UniqueName: \"kubernetes.io/projected/a8a82406-947b-4699-afc0-1794f90d33e7-kube-api-access-258wv\") pod \"a8a82406-947b-4699-afc0-1794f90d33e7\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") "
Jul 30 17:34:37.108137 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-kube-rbac-proxy-sar-config\") pod \"a8a82406-947b-4699-afc0-1794f90d33e7\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") "
Jul 30 17:34:37.108441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108207    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-cabundle-cert\") pod \"a8a82406-947b-4699-afc0-1794f90d33e7\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") "
Jul 30 17:34:37.108441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108251    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a82406-947b-4699-afc0-1794f90d33e7-proxy-tls\") pod \"a8a82406-947b-4699-afc0-1794f90d33e7\" (UID: \"a8a82406-947b-4699-afc0-1794f90d33e7\") "
Jul 30 17:34:37.108558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108424    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a82406-947b-4699-afc0-1794f90d33e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8a82406-947b-4699-afc0-1794f90d33e7" (UID: "a8a82406-947b-4699-afc0-1794f90d33e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:34:37.108558 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108527    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8a82406-947b-4699-afc0-1794f90d33e7" (UID: "a8a82406-947b-4699-afc0-1794f90d33e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:34:37.108669 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.108571    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8a82406-947b-4699-afc0-1794f90d33e7" (UID: "a8a82406-947b-4699-afc0-1794f90d33e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:34:37.110118 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.110098    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a82406-947b-4699-afc0-1794f90d33e7-kube-api-access-258wv" (OuterVolumeSpecName: "kube-api-access-258wv") pod "a8a82406-947b-4699-afc0-1794f90d33e7" (UID: "a8a82406-947b-4699-afc0-1794f90d33e7"). InnerVolumeSpecName "kube-api-access-258wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:34:37.110394 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.110371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a82406-947b-4699-afc0-1794f90d33e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8a82406-947b-4699-afc0-1794f90d33e7" (UID: "a8a82406-947b-4699-afc0-1794f90d33e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:34:37.209483 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.209449    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.209483 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.209482    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a82406-947b-4699-afc0-1794f90d33e7-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.209483 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.209492    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a82406-947b-4699-afc0-1794f90d33e7-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.209724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.209502    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258wv\" (UniqueName: \"kubernetes.io/projected/a8a82406-947b-4699-afc0-1794f90d33e7-kube-api-access-258wv\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.209724 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.209511    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a82406-947b-4699-afc0-1794f90d33e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:34:37.549905 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.549872    2569 generic.go:356] "Generic (PLEG): container finished" podID="a8a82406-947b-4699-afc0-1794f90d33e7" containerID="1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba" exitCode=0
Jul 30 17:34:37.550216 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.549927    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"
Jul 30 17:34:37.550216 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.549963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerDied","Data":"1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba"}
Jul 30 17:34:37.550216 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.550038    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58" event={"ID":"a8a82406-947b-4699-afc0-1794f90d33e7","Type":"ContainerDied","Data":"f001eb320af1e0e1384bf24ad341854a2b5e27c283a25a329aae6f9d29109ef7"}
Jul 30 17:34:37.550216 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.550077    2569 scope.go:117] "RemoveContainer" containerID="c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c"
Jul 30 17:34:37.558856 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.558837    2569 scope.go:117] "RemoveContainer" containerID="1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba"
Jul 30 17:34:37.566269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.566235    2569 scope.go:117] "RemoveContainer" containerID="d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7"
Jul 30 17:34:37.573148 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.573129    2569 scope.go:117] "RemoveContainer" containerID="c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c"
Jul 30 17:34:37.573397 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:34:37.573378    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c\": container with ID starting with c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c not found: ID does not exist" containerID="c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c"
Jul 30 17:34:37.573448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.573406    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c"} err="failed to get container status \"c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c\": rpc error: code = NotFound desc = could not find container \"c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c\": container with ID starting with c7af9616ea614964e137f5810142e8e851c3a7328b58a7f8c19f1bb86912531c not found: ID does not exist"
Jul 30 17:34:37.573448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.573423    2569 scope.go:117] "RemoveContainer" containerID="1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba"
Jul 30 17:34:37.573640 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:34:37.573625    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba\": container with ID starting with 1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba not found: ID does not exist" containerID="1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba"
Jul 30 17:34:37.573682 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.573643    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba"} err="failed to get container status \"1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba\": rpc error: code = NotFound desc = could not find container \"1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba\": container with ID starting with 1de5ab3cc328ff226ff59d6a16d23965f17b0126419e921e560560d0c899d2ba not found: ID does not exist"
Jul 30 17:34:37.573682 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.573656    2569 scope.go:117] "RemoveContainer" containerID="d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7"
Jul 30 17:34:37.573866 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:34:37.573850    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7\": container with ID starting with d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7 not found: ID does not exist" containerID="d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7"
Jul 30 17:34:37.573907 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.573871    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7"} err="failed to get container status \"d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7\": rpc error: code = NotFound desc = could not find container \"d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7\": container with ID starting with d8beed06ee7db6821c06d4185f43b46871ee0215251f72bcfd174dcfa01401f7 not found: ID does not exist"
Jul 30 17:34:37.576997 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.576975    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"]
Jul 30 17:34:37.579854 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:37.579833    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7c9c87bcff-49q58"]
Jul 30 17:34:38.651328 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:34:38.651295    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" path="/var/lib/kubelet/pods/a8a82406-947b-4699-afc0-1794f90d33e7/volumes"
Jul 30 17:45:29.670688 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670612    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"]
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670883    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kube-rbac-proxy"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670895    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kube-rbac-proxy"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670904    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="storage-initializer"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670909    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="storage-initializer"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670924    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670929    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670974    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kube-rbac-proxy"
Jul 30 17:45:29.671152 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.670982    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a82406-947b-4699-afc0-1794f90d33e7" containerName="kserve-container"
Jul 30 17:45:29.674038 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.674019    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.676617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.676574    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:45:29.676736 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.676582    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 30 17:45:29.676736 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.676598    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:45:29.677605 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.677588    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:45:29.677714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.677641    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:45:29.677714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.677660    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:45:29.677814 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.677714    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 30 17:45:29.682519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.682500    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"]
Jul 30 17:45:29.774320 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.774279    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.774320 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.774323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.774555 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.774353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.774555 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.774402    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drpx5\" (UniqueName: \"kubernetes.io/projected/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-api-access-drpx5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.774555 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.774444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.874915 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.874877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.874915 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.874920    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drpx5\" (UniqueName: \"kubernetes.io/projected/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-api-access-drpx5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.875132 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.874956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.875132 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.875019    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.875132 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.875049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.875289 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:45:29.875155    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 30 17:45:29.875289 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:45:29.875236    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls podName:ade0ba40-16bb-4bb8-9be0-f9265ae4cc83 nodeName:}" failed. No retries permitted until 2026-07-30 17:45:30.375212365 +0000 UTC m=+2698.408022681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-gd5z7" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 30 17:45:29.875449 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.875426    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.875635 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.875619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.875689 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.875672    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:29.884302 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:29.884275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drpx5\" (UniqueName: \"kubernetes.io/projected/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-api-access-drpx5\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:30.379728 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:30.379674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:30.379924 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:45:30.379836    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 30 17:45:30.379924 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:45:30.379918    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls podName:ade0ba40-16bb-4bb8-9be0-f9265ae4cc83 nodeName:}" failed. No retries permitted until 2026-07-30 17:45:31.379899389 +0000 UTC m=+2699.412709705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-gd5z7" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 30 17:45:31.387308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:31.387242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:31.389703 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:31.389681    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-gd5z7\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:31.485391 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:31.485364    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:31.613033 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:31.613005    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"]
Jul 30 17:45:31.616182 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:45:31.616149    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade0ba40_16bb_4bb8_9be0_f9265ae4cc83.slice/crio-612259c776c7b5d438142ff5f31c4058f6dbe9f866bc8b00e7637b3d1e319907 WatchSource:0}: Error finding container 612259c776c7b5d438142ff5f31c4058f6dbe9f866bc8b00e7637b3d1e319907: Status 404 returned error can't find the container with id 612259c776c7b5d438142ff5f31c4058f6dbe9f866bc8b00e7637b3d1e319907
Jul 30 17:45:31.899669 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:31.899580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerStarted","Data":"e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11"}
Jul 30 17:45:31.899669 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:31.899639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerStarted","Data":"612259c776c7b5d438142ff5f31c4058f6dbe9f866bc8b00e7637b3d1e319907"}
Jul 30 17:45:36.918093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:36.918005    2569 generic.go:356] "Generic (PLEG): container finished" podID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerID="e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11" exitCode=0
Jul 30 17:45:36.918502 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:36.918076    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerDied","Data":"e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11"}
Jul 30 17:45:36.919310 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:36.919294    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:45:49.975013 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:49.974975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerStarted","Data":"6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67"}
Jul 30 17:45:49.975420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:49.975069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerStarted","Data":"a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d"}
Jul 30 17:45:49.975420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:49.975293    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:49.999035 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:49.998983    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podStartSLOduration=3.757296483 podStartE2EDuration="20.998967186s" totalImagesPullingTime="12.383696643s" totalInitContainerRuntime="4.85797406s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:45:49.997017368 +0000 UTC m=+2718.029827704" watchObservedRunningTime="2026-07-30 17:45:49.998967186 +0000 UTC m=+2718.031777523"
Jul 30 17:45:50.979206 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:50.979173    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:50.980456 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:50.980428    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:45:51.983120 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:51.983079    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:45:56.988228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:56.988195    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:45:56.988710 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:45:56.988682    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:46:06.989268 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:06.989207    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:46:16.989488 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:16.989449    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:46:26.989368 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:26.989328    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 30 17:46:36.989454 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:36.989418    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:46:41.289672 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:41.289634    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"]
Jul 30 17:46:41.290037 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:41.289846    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container" containerID="cri-o://a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d" gracePeriod=30
Jul 30 17:46:41.290037 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:41.289926    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kube-rbac-proxy" containerID="cri-o://6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67" gracePeriod=30
Jul 30 17:46:41.983711 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:41.983648    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 30 17:46:42.170079 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:42.170045    2569 generic.go:356] "Generic (PLEG): container finished" podID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerID="6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67" exitCode=2
Jul 30 17:46:42.170241 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:42.170094    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerDied","Data":"6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67"}
Jul 30 17:46:44.031609 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.031582    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:46:44.056906 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.056873    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-cabundle-cert\") pod \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") "
Jul 30 17:46:44.057074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.056919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kserve-provision-location\") pod \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") "
Jul 30 17:46:44.057074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.056996    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") pod \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") "
Jul 30 17:46:44.057074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.057019    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drpx5\" (UniqueName: \"kubernetes.io/projected/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-api-access-drpx5\") pod \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") "
Jul 30 17:46:44.057074 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.057046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-rbac-proxy-sar-config\") pod \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\" (UID: \"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83\") "
Jul 30 17:46:44.057321 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.057291    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:46:44.057776 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.057654    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:46:44.059904 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.059879    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-api-access-drpx5" (OuterVolumeSpecName: "kube-api-access-drpx5") pod "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83"). InnerVolumeSpecName "kube-api-access-drpx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:46:44.060250 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.060223    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:46:44.068533 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.068502    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" (UID: "ade0ba40-16bb-4bb8-9be0-f9265ae4cc83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:46:44.157932 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.157871    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:46:44.157932 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.157896    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drpx5\" (UniqueName: \"kubernetes.io/projected/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-api-access-drpx5\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:46:44.157932 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.157905    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:46:44.157932 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.157914    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:46:44.157932 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.157923    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:46:44.179199 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.179173    2569 generic.go:356] "Generic (PLEG): container finished" podID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerID="a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d" exitCode=0
Jul 30 17:46:44.179344 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.179208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerDied","Data":"a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d"}
Jul 30 17:46:44.179344 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.179242    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"
Jul 30 17:46:44.179344 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.179295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7" event={"ID":"ade0ba40-16bb-4bb8-9be0-f9265ae4cc83","Type":"ContainerDied","Data":"612259c776c7b5d438142ff5f31c4058f6dbe9f866bc8b00e7637b3d1e319907"}
Jul 30 17:46:44.179344 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.179327    2569 scope.go:117] "RemoveContainer" containerID="6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67"
Jul 30 17:46:44.187747 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.187727    2569 scope.go:117] "RemoveContainer" containerID="a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d"
Jul 30 17:46:44.194786 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.194768    2569 scope.go:117] "RemoveContainer" containerID="e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11"
Jul 30 17:46:44.201384 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.201368    2569 scope.go:117] "RemoveContainer" containerID="6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67"
Jul 30 17:46:44.201607 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:46:44.201591    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67\": container with ID starting with 6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67 not found: ID does not exist" containerID="6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67"
Jul 30 17:46:44.201668 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.201614    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67"} err="failed to get container status \"6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67\": rpc error: code = NotFound desc = could not find container \"6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67\": container with ID starting with 6a8db095b921dec6f9cdfc4a92b57db9a736c24c3e0d1de1cfc2174625622d67 not found: ID does not exist"
Jul 30 17:46:44.201668 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.201628    2569 scope.go:117] "RemoveContainer" containerID="a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d"
Jul 30 17:46:44.201828 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:46:44.201814    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d\": container with ID starting with a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d not found: ID does not exist" containerID="a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d"
Jul 30 17:46:44.201868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.201848    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d"} err="failed to get container status \"a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d\": rpc error: code = NotFound desc = could not find container \"a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d\": container with ID starting with a885a608a3f473e4bc02e2567e9225602c64ca147eea2ea6dd57020f01d5c31d not found: ID does not exist"
Jul 30 17:46:44.201868 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.201861    2569 scope.go:117] "RemoveContainer" containerID="e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11"
Jul 30 17:46:44.202083 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:46:44.202065    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11\": container with ID starting with e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11 not found: ID does not exist" containerID="e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11"
Jul 30 17:46:44.202136 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.202091    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11"} err="failed to get container status \"e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11\": rpc error: code = NotFound desc = could not find container \"e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11\": container with ID starting with e5c0bbdfc5fa75a3efac568688afd5d209f693dcf54031f11e2d5ef183fefe11 not found: ID does not exist"
Jul 30 17:46:44.210251 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.210228    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"]
Jul 30 17:46:44.216745 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.216724    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-gd5z7"]
Jul 30 17:46:44.651510 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:46:44.651477    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" path="/var/lib/kubelet/pods/ade0ba40-16bb-4bb8-9be0-f9265ae4cc83/volumes"
Jul 30 17:48:03.233455 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233416    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"]
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233682    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233694    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233704    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="storage-initializer"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233710    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="storage-initializer"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233725    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kube-rbac-proxy"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233731    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kube-rbac-proxy"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233773    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kube-rbac-proxy"
Jul 30 17:48:03.233925 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.233782    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade0ba40-16bb-4bb8-9be0-f9265ae4cc83" containerName="kserve-container"
Jul 30 17:48:03.235663 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.235647    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.238956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.238928    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 30 17:48:03.239109 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.238976    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:48:03.239109 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.239001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:48:03.239109 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.239026    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:48:03.239740 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.239723    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:48:03.239835 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.239745    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 30 17:48:03.240343 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.240328    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:48:03.245056 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.245021    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.245163 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.245099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.245163 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.245146    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00853d34-09d6-412b-9037-d0d384389404-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.245297 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.245190    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669k9\" (UniqueName: \"kubernetes.io/projected/00853d34-09d6-412b-9037-d0d384389404-kube-api-access-669k9\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.245297 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.245220    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.246738 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.246718    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"]
Jul 30 17:48:03.345962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.345923    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669k9\" (UniqueName: \"kubernetes.io/projected/00853d34-09d6-412b-9037-d0d384389404-kube-api-access-669k9\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.345962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.345961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.346226 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.345992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.346226 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.346037    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.346226 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.346063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00853d34-09d6-412b-9037-d0d384389404-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.346226 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:48:03.346185    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 30 17:48:03.346402 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:48:03.346283    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls podName:00853d34-09d6-412b-9037-d0d384389404 nodeName:}" failed. No retries permitted until 2026-07-30 17:48:03.846239025 +0000 UTC m=+2851.879049339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" (UID: "00853d34-09d6-412b-9037-d0d384389404") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 30 17:48:03.346560 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.346537    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00853d34-09d6-412b-9037-d0d384389404-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.346730 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.346712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.346766 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.346721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.356157 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.356129    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-669k9\" (UniqueName: \"kubernetes.io/projected/00853d34-09d6-412b-9037-d0d384389404-kube-api-access-669k9\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.850090 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.850040    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:03.852427 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:03.852407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:04.146982 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:04.146883    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:04.280372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:04.280344    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"]
Jul 30 17:48:04.282925 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:48:04.282896    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00853d34_09d6_412b_9037_d0d384389404.slice/crio-fe79c39b9f9cb2111c0db4d6a82e01e42d64bbfb65c169f4eb016eba7beb48e7 WatchSource:0}: Error finding container fe79c39b9f9cb2111c0db4d6a82e01e42d64bbfb65c169f4eb016eba7beb48e7: Status 404 returned error can't find the container with id fe79c39b9f9cb2111c0db4d6a82e01e42d64bbfb65c169f4eb016eba7beb48e7
Jul 30 17:48:04.468744 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:04.468705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerStarted","Data":"3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db"}
Jul 30 17:48:04.468947 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:04.468766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerStarted","Data":"fe79c39b9f9cb2111c0db4d6a82e01e42d64bbfb65c169f4eb016eba7beb48e7"}
Jul 30 17:48:09.489583 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:09.489545    2569 generic.go:356] "Generic (PLEG): container finished" podID="00853d34-09d6-412b-9037-d0d384389404" containerID="3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db" exitCode=0
Jul 30 17:48:09.490120 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:09.489622    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerDied","Data":"3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db"}
Jul 30 17:48:10.495644 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:10.495613    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerStarted","Data":"4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd"}
Jul 30 17:48:10.496018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:10.495675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerStarted","Data":"9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d"}
Jul 30 17:48:10.496018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:10.495943    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:10.496018 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:10.495999    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:10.497285 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:10.497242    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:48:10.519578 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:10.519533    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podStartSLOduration=2.935493622 podStartE2EDuration="7.519518926s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.584025304s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:48:10.518336721 +0000 UTC m=+2858.551147060" watchObservedRunningTime="2026-07-30 17:48:10.519518926 +0000 UTC m=+2858.552329263"
Jul 30 17:48:11.500092 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:11.500052    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:48:16.505176 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:16.505142    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:48:16.505809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:16.505779    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:48:26.506540 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:26.506451    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:48:36.506608 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:36.506565    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:48:46.506370 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:46.506329    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:48:56.506740 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:48:56.506707    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:49:05.177850 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:05.177815    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"]
Jul 30 17:49:05.178376 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:05.178053    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" containerID="cri-o://9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d" gracePeriod=30
Jul 30 17:49:05.178376 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:05.178100    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kube-rbac-proxy" containerID="cri-o://4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd" gracePeriod=30
Jul 30 17:49:05.696382 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:05.696345    2569 generic.go:356] "Generic (PLEG): container finished" podID="00853d34-09d6-412b-9037-d0d384389404" containerID="4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd" exitCode=2
Jul 30 17:49:05.696554 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:05.696422    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerDied","Data":"4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd"}
Jul 30 17:49:06.500541 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:06.500497    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 30 17:49:06.505852 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:06.505828    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 30 17:49:07.920048 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.920020    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:49:07.964092 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964061    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-669k9\" (UniqueName: \"kubernetes.io/projected/00853d34-09d6-412b-9037-d0d384389404-kube-api-access-669k9\") pod \"00853d34-09d6-412b-9037-d0d384389404\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") "
Jul 30 17:49:07.964269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964137    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls\") pod \"00853d34-09d6-412b-9037-d0d384389404\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") "
Jul 30 17:49:07.964269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964162    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-kube-rbac-proxy-sar-config\") pod \"00853d34-09d6-412b-9037-d0d384389404\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") "
Jul 30 17:49:07.964269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964181    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-cabundle-cert\") pod \"00853d34-09d6-412b-9037-d0d384389404\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") "
Jul 30 17:49:07.964269 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964219    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00853d34-09d6-412b-9037-d0d384389404-kserve-provision-location\") pod \"00853d34-09d6-412b-9037-d0d384389404\" (UID: \"00853d34-09d6-412b-9037-d0d384389404\") "
Jul 30 17:49:07.964589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964559    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00853d34-09d6-412b-9037-d0d384389404" (UID: "00853d34-09d6-412b-9037-d0d384389404"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:49:07.964589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.964567    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00853d34-09d6-412b-9037-d0d384389404" (UID: "00853d34-09d6-412b-9037-d0d384389404"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:49:07.966228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.966201    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00853d34-09d6-412b-9037-d0d384389404-kube-api-access-669k9" (OuterVolumeSpecName: "kube-api-access-669k9") pod "00853d34-09d6-412b-9037-d0d384389404" (UID: "00853d34-09d6-412b-9037-d0d384389404"). InnerVolumeSpecName "kube-api-access-669k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:49:07.966228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.966220    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00853d34-09d6-412b-9037-d0d384389404" (UID: "00853d34-09d6-412b-9037-d0d384389404"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:49:07.974020 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:07.973992    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00853d34-09d6-412b-9037-d0d384389404-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00853d34-09d6-412b-9037-d0d384389404" (UID: "00853d34-09d6-412b-9037-d0d384389404"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:49:08.065600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.065516    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00853d34-09d6-412b-9037-d0d384389404-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:49:08.065600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.065545    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:49:08.065600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.065555    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00853d34-09d6-412b-9037-d0d384389404-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:49:08.065600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.065564    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00853d34-09d6-412b-9037-d0d384389404-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:49:08.065600 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.065572    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-669k9\" (UniqueName: \"kubernetes.io/projected/00853d34-09d6-412b-9037-d0d384389404-kube-api-access-669k9\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:49:08.709489 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.709448    2569 generic.go:356] "Generic (PLEG): container finished" podID="00853d34-09d6-412b-9037-d0d384389404" containerID="9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d" exitCode=0
Jul 30 17:49:08.709676 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.709516    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"
Jul 30 17:49:08.709676 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.709524    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerDied","Data":"9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d"}
Jul 30 17:49:08.709676 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.709588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g" event={"ID":"00853d34-09d6-412b-9037-d0d384389404","Type":"ContainerDied","Data":"fe79c39b9f9cb2111c0db4d6a82e01e42d64bbfb65c169f4eb016eba7beb48e7"}
Jul 30 17:49:08.709676 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.709615    2569 scope.go:117] "RemoveContainer" containerID="4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd"
Jul 30 17:49:08.717901 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.717877    2569 scope.go:117] "RemoveContainer" containerID="9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d"
Jul 30 17:49:08.725969 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.725950    2569 scope.go:117] "RemoveContainer" containerID="3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db"
Jul 30 17:49:08.731374 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.731349    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"]
Jul 30 17:49:08.733871 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.733847    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ml44g"]
Jul 30 17:49:08.734308 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.734282    2569 scope.go:117] "RemoveContainer" containerID="4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd"
Jul 30 17:49:08.734568 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:49:08.734548    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd\": container with ID starting with 4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd not found: ID does not exist" containerID="4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd"
Jul 30 17:49:08.734656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.734575    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd"} err="failed to get container status \"4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd\": rpc error: code = NotFound desc = could not find container \"4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd\": container with ID starting with 4e2338548d4e50a10c1ac731d4355524657d1f74f6ee8c2bcc32467676897afd not found: ID does not exist"
Jul 30 17:49:08.734656 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.734592    2569 scope.go:117] "RemoveContainer" containerID="9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d"
Jul 30 17:49:08.734868 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:49:08.734848    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d\": container with ID starting with 9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d not found: ID does not exist" containerID="9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d"
Jul 30 17:49:08.734944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.734873    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d"} err="failed to get container status \"9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d\": rpc error: code = NotFound desc = could not find container \"9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d\": container with ID starting with 9eefea6e32757fb9b41e6b8fd5f7a90b9976bcef671c3b9a636245f0fc4e0d1d not found: ID does not exist"
Jul 30 17:49:08.734944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.734889    2569 scope.go:117] "RemoveContainer" containerID="3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db"
Jul 30 17:49:08.735125 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:49:08.735105    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db\": container with ID starting with 3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db not found: ID does not exist" containerID="3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db"
Jul 30 17:49:08.735186 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:08.735139    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db"} err="failed to get container status \"3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db\": rpc error: code = NotFound desc = could not find container \"3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db\": container with ID starting with 3f78b33de0e8913f47b5372d2a9ab739f5e4cf70138bbe3eacdf1b5a354045db not found: ID does not exist"
Jul 30 17:49:10.353401 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353361    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"]
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353654    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kube-rbac-proxy"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353667    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kube-rbac-proxy"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353683    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353688    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353703    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="storage-initializer"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353708    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="storage-initializer"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353758    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kserve-container"
Jul 30 17:49:10.353791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.353767    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00853d34-09d6-412b-9037-d0d384389404" containerName="kube-rbac-proxy"
Jul 30 17:49:10.358282 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.358246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.361058 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.361035    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 30 17:49:10.361058 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.361059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:49:10.361241 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.361090    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:49:10.361446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.361429    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:49:10.362155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.362132    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:49:10.362155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.362148    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:49:10.362155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.362140    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 30 17:49:10.367395 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.367371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"]
Jul 30 17:49:10.381109 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.381082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.381196 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.381116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvgph\" (UniqueName: \"kubernetes.io/projected/834ab665-bc91-483a-90d1-169de9306e14-kube-api-access-dvgph\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.381196 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.381140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834ab665-bc91-483a-90d1-169de9306e14-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.381196 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.381174    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.381328 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.381226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.481631 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.481588    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.481631 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.481632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvgph\" (UniqueName: \"kubernetes.io/projected/834ab665-bc91-483a-90d1-169de9306e14-kube-api-access-dvgph\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.481881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.481656    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834ab665-bc91-483a-90d1-169de9306e14-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.481881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.481690    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.481881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.481737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.482043 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:49:10.481886    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 30 17:49:10.482043 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:49:10.481974    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls podName:834ab665-bc91-483a-90d1-169de9306e14 nodeName:}" failed. No retries permitted until 2026-07-30 17:49:10.981951877 +0000 UTC m=+2919.014762192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls") pod "isvc-pmml-predictor-8674795f59-vk474" (UID: "834ab665-bc91-483a-90d1-169de9306e14") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 30 17:49:10.482158 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.482092    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834ab665-bc91-483a-90d1-169de9306e14-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.482390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.482372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.482438 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.482372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.491105 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.491082    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvgph\" (UniqueName: \"kubernetes.io/projected/834ab665-bc91-483a-90d1-169de9306e14-kube-api-access-dvgph\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.651818 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.651736    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00853d34-09d6-412b-9037-d0d384389404" path="/var/lib/kubelet/pods/00853d34-09d6-412b-9037-d0d384389404/volumes"
Jul 30 17:49:10.986672 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.986637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:10.989080 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:10.989059    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-vk474\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:11.268166 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:11.268076    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:11.397352 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:11.397320    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"]
Jul 30 17:49:11.400568 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:49:11.400538    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834ab665_bc91_483a_90d1_169de9306e14.slice/crio-8ea71a65cf7c874a466ba745bb3791cff9aa7b2ed955b3426a7fac95b830b620 WatchSource:0}: Error finding container 8ea71a65cf7c874a466ba745bb3791cff9aa7b2ed955b3426a7fac95b830b620: Status 404 returned error can't find the container with id 8ea71a65cf7c874a466ba745bb3791cff9aa7b2ed955b3426a7fac95b830b620
Jul 30 17:49:11.723689 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:11.723653    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerStarted","Data":"5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598"}
Jul 30 17:49:11.723872 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:11.723711    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerStarted","Data":"8ea71a65cf7c874a466ba745bb3791cff9aa7b2ed955b3426a7fac95b830b620"}
Jul 30 17:49:15.739714 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:15.739676    2569 generic.go:356] "Generic (PLEG): container finished" podID="834ab665-bc91-483a-90d1-169de9306e14" containerID="5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598" exitCode=0
Jul 30 17:49:15.740071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:15.739743    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerDied","Data":"5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598"}
Jul 30 17:49:22.773082 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:22.773042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerStarted","Data":"3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21"}
Jul 30 17:49:22.773448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:22.773110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerStarted","Data":"d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2"}
Jul 30 17:49:22.773448 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:22.773315    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:22.794240 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:22.794138    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podStartSLOduration=2.014088034 podStartE2EDuration="12.794125768s" totalImagesPullingTime="6.82867488s" totalInitContainerRuntime="3.951362854s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:49:22.793655596 +0000 UTC m=+2930.826465911" watchObservedRunningTime="2026-07-30 17:49:22.794125768 +0000 UTC m=+2930.826936105"
Jul 30 17:49:23.777381 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:23.777343    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:23.778572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:23.778546    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:49:24.781001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:24.780961    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:49:29.786486 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:29.786454    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:49:29.786990 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:29.786961    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:49:39.787078 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:39.787040    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:49:49.787752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:49.787709    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:49:59.787687 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:49:59.787609    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:50:09.787717 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:09.787672    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:50:19.787466 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:19.787422    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:50:29.787446 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:29.787405    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:50:39.787504 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:39.787464    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:50:44.651679 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:44.651648    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:50:51.404669 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:51.404630    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"]
Jul 30 17:50:51.405151 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:51.404882    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" containerID="cri-o://d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2" gracePeriod=30
Jul 30 17:50:51.405151 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:51.404924    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kube-rbac-proxy" containerID="cri-o://3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21" gracePeriod=30
Jul 30 17:50:52.104963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:52.104924    2569 generic.go:356] "Generic (PLEG): container finished" podID="834ab665-bc91-483a-90d1-169de9306e14" containerID="3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21" exitCode=2
Jul 30 17:50:52.105143 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:52.104995    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerDied","Data":"3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21"}
Jul 30 17:50:54.647417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:54.647373    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 30 17:50:54.781754 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:54.781715    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 30 17:50:55.048159 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.048132    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:50:55.118372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.118328    2569 generic.go:356] "Generic (PLEG): container finished" podID="834ab665-bc91-483a-90d1-169de9306e14" containerID="d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2" exitCode=0
Jul 30 17:50:55.118568 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.118377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerDied","Data":"d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2"}
Jul 30 17:50:55.118568 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.118386    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"
Jul 30 17:50:55.118568 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.118434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474" event={"ID":"834ab665-bc91-483a-90d1-169de9306e14","Type":"ContainerDied","Data":"8ea71a65cf7c874a466ba745bb3791cff9aa7b2ed955b3426a7fac95b830b620"}
Jul 30 17:50:55.118568 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.118467    2569 scope.go:117] "RemoveContainer" containerID="3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21"
Jul 30 17:50:55.126190 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.126173    2569 scope.go:117] "RemoveContainer" containerID="d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2"
Jul 30 17:50:55.132808 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.132785    2569 scope.go:117] "RemoveContainer" containerID="5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598"
Jul 30 17:50:55.139322 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.139304    2569 scope.go:117] "RemoveContainer" containerID="3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21"
Jul 30 17:50:55.139572 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:50:55.139556    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21\": container with ID starting with 3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21 not found: ID does not exist" containerID="3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21"
Jul 30 17:50:55.139637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.139580    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21"} err="failed to get container status \"3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21\": rpc error: code = NotFound desc = could not find container \"3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21\": container with ID starting with 3d64dedfb9998925bee47e277ad2e3d2da52ff1dd5cf35eed8a0d761a327ef21 not found: ID does not exist"
Jul 30 17:50:55.139637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.139596    2569 scope.go:117] "RemoveContainer" containerID="d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2"
Jul 30 17:50:55.139830 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:50:55.139816    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2\": container with ID starting with d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2 not found: ID does not exist" containerID="d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2"
Jul 30 17:50:55.139888 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.139833    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2"} err="failed to get container status \"d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2\": rpc error: code = NotFound desc = could not find container \"d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2\": container with ID starting with d206b8983482def2f719025de3c84110e330d2bc88935b4ff6d49647c2a9a3a2 not found: ID does not exist"
Jul 30 17:50:55.139888 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.139845    2569 scope.go:117] "RemoveContainer" containerID="5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598"
Jul 30 17:50:55.140066 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:50:55.140042    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598\": container with ID starting with 5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598 not found: ID does not exist" containerID="5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598"
Jul 30 17:50:55.140123 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.140077    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598"} err="failed to get container status \"5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598\": rpc error: code = NotFound desc = could not find container \"5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598\": container with ID starting with 5e22ab7eb0ee2255efc08c1416faa8ef0f114a82fe8485da3da46f05e6568598 not found: ID does not exist"
Jul 30 17:50:55.207519 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207498    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-kube-rbac-proxy-sar-config\") pod \"834ab665-bc91-483a-90d1-169de9306e14\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") "
Jul 30 17:50:55.207657 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207542    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvgph\" (UniqueName: \"kubernetes.io/projected/834ab665-bc91-483a-90d1-169de9306e14-kube-api-access-dvgph\") pod \"834ab665-bc91-483a-90d1-169de9306e14\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") "
Jul 30 17:50:55.207657 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834ab665-bc91-483a-90d1-169de9306e14-kserve-provision-location\") pod \"834ab665-bc91-483a-90d1-169de9306e14\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") "
Jul 30 17:50:55.207780 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207678    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls\") pod \"834ab665-bc91-483a-90d1-169de9306e14\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") "
Jul 30 17:50:55.207780 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207706    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-cabundle-cert\") pod \"834ab665-bc91-483a-90d1-169de9306e14\" (UID: \"834ab665-bc91-483a-90d1-169de9306e14\") "
Jul 30 17:50:55.207892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207844    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834ab665-bc91-483a-90d1-169de9306e14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "834ab665-bc91-483a-90d1-169de9306e14" (UID: "834ab665-bc91-483a-90d1-169de9306e14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:50:55.207892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207874    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "834ab665-bc91-483a-90d1-169de9306e14" (UID: "834ab665-bc91-483a-90d1-169de9306e14"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:50:55.207980 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207950    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:50:55.207980 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.207961    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834ab665-bc91-483a-90d1-169de9306e14-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:50:55.208178 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.208153    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "834ab665-bc91-483a-90d1-169de9306e14" (UID: "834ab665-bc91-483a-90d1-169de9306e14"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:50:55.209574 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.209547    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "834ab665-bc91-483a-90d1-169de9306e14" (UID: "834ab665-bc91-483a-90d1-169de9306e14"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:50:55.209674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.209612    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834ab665-bc91-483a-90d1-169de9306e14-kube-api-access-dvgph" (OuterVolumeSpecName: "kube-api-access-dvgph") pod "834ab665-bc91-483a-90d1-169de9306e14" (UID: "834ab665-bc91-483a-90d1-169de9306e14"). InnerVolumeSpecName "kube-api-access-dvgph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:50:55.309195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.309162    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvgph\" (UniqueName: \"kubernetes.io/projected/834ab665-bc91-483a-90d1-169de9306e14-kube-api-access-dvgph\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:50:55.309195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.309186    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/834ab665-bc91-483a-90d1-169de9306e14-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:50:55.309195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.309199    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834ab665-bc91-483a-90d1-169de9306e14-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:50:55.444754 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.444725    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"]
Jul 30 17:50:55.447488 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:55.447463    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-vk474"]
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.557538    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"]
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558218    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="storage-initializer"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558238    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="storage-initializer"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558280    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kube-rbac-proxy"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558290    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kube-rbac-proxy"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558305    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558314    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558451    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kserve-container"
Jul 30 17:50:56.560301 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.558460    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="834ab665-bc91-483a-90d1-169de9306e14" containerName="kube-rbac-proxy"
Jul 30 17:50:56.563694 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.563662    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.566357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.566327    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:50:56.566520 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.566481    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:50:56.566520 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.566492    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 30 17:50:56.566642 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.566626    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:50:56.566707 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.566664    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:50:56.567420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.567399    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 30 17:50:56.567510 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.567409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:50:56.573323 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.573298    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"]
Jul 30 17:50:56.651616 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.651577    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834ab665-bc91-483a-90d1-169de9306e14" path="/var/lib/kubelet/pods/834ab665-bc91-483a-90d1-169de9306e14/volumes"
Jul 30 17:50:56.720615 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.720568    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97bh\" (UniqueName: \"kubernetes.io/projected/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-api-access-z97bh\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.720812 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.720635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.720812 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.720674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.720812 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.720713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e86bd96-77d8-411c-9dbf-f01d951d57fc-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.720812 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.720735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.821366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.821276    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.821366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.821325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.821366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.821356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e86bd96-77d8-411c-9dbf-f01d951d57fc-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.821631 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.821371    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.821631 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.821447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97bh\" (UniqueName: \"kubernetes.io/projected/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-api-access-z97bh\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.821779 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.821728    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.822061 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.822038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.822153 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.822083    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.824061 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.824044    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e86bd96-77d8-411c-9dbf-f01d951d57fc-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.831653 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.831632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97bh\" (UniqueName: \"kubernetes.io/projected/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-api-access-z97bh\") pod \"isvc-pmml-runtime-predictor-885fc8777-xf2vs\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:56.874916 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:56.874883    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:50:57.005457 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:57.005432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"]
Jul 30 17:50:57.007927 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:50:57.007903    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e86bd96_77d8_411c_9dbf_f01d951d57fc.slice/crio-7da4ad05918eadaf51765b15df30d3668fb9da269409bbc26da9444890560837 WatchSource:0}: Error finding container 7da4ad05918eadaf51765b15df30d3668fb9da269409bbc26da9444890560837: Status 404 returned error can't find the container with id 7da4ad05918eadaf51765b15df30d3668fb9da269409bbc26da9444890560837
Jul 30 17:50:57.128624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:57.128542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerStarted","Data":"e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4"}
Jul 30 17:50:57.128624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:50:57.128604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerStarted","Data":"7da4ad05918eadaf51765b15df30d3668fb9da269409bbc26da9444890560837"}
Jul 30 17:51:01.145245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:01.145156    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerID="e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4" exitCode=0
Jul 30 17:51:01.145245 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:01.145236    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerDied","Data":"e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4"}
Jul 30 17:51:02.151271 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:02.151225    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerStarted","Data":"ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2"}
Jul 30 17:51:02.151674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:02.151297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerStarted","Data":"5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755"}
Jul 30 17:51:02.151674 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:02.151489    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:51:02.171634 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:02.171576    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podStartSLOduration=2.486757169 podStartE2EDuration="6.171557493s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684800324s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:51:02.171125253 +0000 UTC m=+3030.203935590" watchObservedRunningTime="2026-07-30 17:51:02.171557493 +0000 UTC m=+3030.204367839"
Jul 30 17:51:03.154654 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:03.154614    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:51:03.156003 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:03.155976    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:04.158458 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:04.158418    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:09.162967 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:09.162938    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:51:09.163589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:09.163561    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:19.163751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:19.163712    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:29.163792 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:29.163704    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:39.163755 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:39.163710    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:49.163817 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:49.163776    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:51:59.163582 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:51:59.163538    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:52:09.163938 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:09.163892    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:52:19.163642 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:19.163600    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:52:19.647862 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:19.647826    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:52:27.497746 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:27.497711    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"]
Jul 30 17:52:27.498188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:27.497923    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" containerID="cri-o://5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755" gracePeriod=30
Jul 30 17:52:27.498188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:27.497976    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kube-rbac-proxy" containerID="cri-o://ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2" gracePeriod=30
Jul 30 17:52:28.458380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:28.458349    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerID="ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2" exitCode=2
Jul 30 17:52:28.458548 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:28.458421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerDied","Data":"ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2"}
Jul 30 17:52:29.159203 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:29.159159    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 30 17:52:29.647548 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:29.647505    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 30 17:52:31.040920 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.040893    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:52:31.056763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.056729    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kserve-provision-location\") pod \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") "
Jul 30 17:52:31.056763 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.056769    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-cabundle-cert\") pod \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") "
Jul 30 17:52:31.056958 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.056794    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e86bd96-77d8-411c-9dbf-f01d951d57fc-proxy-tls\") pod \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") "
Jul 30 17:52:31.056958 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.056919    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z97bh\" (UniqueName: \"kubernetes.io/projected/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-api-access-z97bh\") pod \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") "
Jul 30 17:52:31.057071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.056983    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-rbac-proxy-sar-config\") pod \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\" (UID: \"3e86bd96-77d8-411c-9dbf-f01d951d57fc\") "
Jul 30 17:52:31.057071 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.057049    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e86bd96-77d8-411c-9dbf-f01d951d57fc" (UID: "3e86bd96-77d8-411c-9dbf-f01d951d57fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:52:31.057296 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.057247    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e86bd96-77d8-411c-9dbf-f01d951d57fc" (UID: "3e86bd96-77d8-411c-9dbf-f01d951d57fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:52:31.057434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.057351    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:52:31.057434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.057353    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e86bd96-77d8-411c-9dbf-f01d951d57fc" (UID: "3e86bd96-77d8-411c-9dbf-f01d951d57fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:52:31.057434 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.057371    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:52:31.058941 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.058922    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e86bd96-77d8-411c-9dbf-f01d951d57fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e86bd96-77d8-411c-9dbf-f01d951d57fc" (UID: "3e86bd96-77d8-411c-9dbf-f01d951d57fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:52:31.059048 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.059031    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-api-access-z97bh" (OuterVolumeSpecName: "kube-api-access-z97bh") pod "3e86bd96-77d8-411c-9dbf-f01d951d57fc" (UID: "3e86bd96-77d8-411c-9dbf-f01d951d57fc"). InnerVolumeSpecName "kube-api-access-z97bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:52:31.158149 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.158043    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e86bd96-77d8-411c-9dbf-f01d951d57fc-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:52:31.158149 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.158091    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97bh\" (UniqueName: \"kubernetes.io/projected/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-api-access-z97bh\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:52:31.158149 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.158105    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e86bd96-77d8-411c-9dbf-f01d951d57fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:52:31.475708 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.475675    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerID="5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755" exitCode=0
Jul 30 17:52:31.475881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.475725    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerDied","Data":"5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755"}
Jul 30 17:52:31.475881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.475772    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"
Jul 30 17:52:31.475881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.475794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs" event={"ID":"3e86bd96-77d8-411c-9dbf-f01d951d57fc","Type":"ContainerDied","Data":"7da4ad05918eadaf51765b15df30d3668fb9da269409bbc26da9444890560837"}
Jul 30 17:52:31.475881 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.475819    2569 scope.go:117] "RemoveContainer" containerID="ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2"
Jul 30 17:52:31.483985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.483738    2569 scope.go:117] "RemoveContainer" containerID="5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755"
Jul 30 17:52:31.490636 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.490618    2569 scope.go:117] "RemoveContainer" containerID="e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4"
Jul 30 17:52:31.497597 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.497575    2569 scope.go:117] "RemoveContainer" containerID="ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2"
Jul 30 17:52:31.497860 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:52:31.497841    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2\": container with ID starting with ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2 not found: ID does not exist" containerID="ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2"
Jul 30 17:52:31.497921 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.497867    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2"} err="failed to get container status \"ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2\": rpc error: code = NotFound desc = could not find container \"ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2\": container with ID starting with ab4f310bac0acce56f65c51446d37fe2f04f6dc9fc7256c842e4d28e418d31f2 not found: ID does not exist"
Jul 30 17:52:31.497921 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.497884    2569 scope.go:117] "RemoveContainer" containerID="5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755"
Jul 30 17:52:31.498164 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:52:31.498146    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755\": container with ID starting with 5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755 not found: ID does not exist" containerID="5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755"
Jul 30 17:52:31.498219 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.498170    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755"} err="failed to get container status \"5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755\": rpc error: code = NotFound desc = could not find container \"5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755\": container with ID starting with 5d0060c487bc0dfdd968374e7405079df183fa83b7175d138c1c6b5c066c4755 not found: ID does not exist"
Jul 30 17:52:31.498219 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.498184    2569 scope.go:117] "RemoveContainer" containerID="e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4"
Jul 30 17:52:31.498429 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:52:31.498410    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4\": container with ID starting with e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4 not found: ID does not exist" containerID="e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4"
Jul 30 17:52:31.498477 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.498434    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4"} err="failed to get container status \"e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4\": rpc error: code = NotFound desc = could not find container \"e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4\": container with ID starting with e9468d5886c936cffd7840c0546f634b4e466d512270408733c2f2f97ac7fdf4 not found: ID does not exist"
Jul 30 17:52:31.498892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.498875    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"]
Jul 30 17:52:31.502975 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:31.502956    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xf2vs"]
Jul 30 17:52:32.652210 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.652182    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" path="/var/lib/kubelet/pods/3e86bd96-77d8-411c-9dbf-f01d951d57fc/volumes"
Jul 30 17:52:32.671315 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671282    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"]
Jul 30 17:52:32.671577 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671564    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="storage-initializer"
Jul 30 17:52:32.671621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671579    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="storage-initializer"
Jul 30 17:52:32.671621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671588    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container"
Jul 30 17:52:32.671621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671593    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container"
Jul 30 17:52:32.671621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671605    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kube-rbac-proxy"
Jul 30 17:52:32.671621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671611    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kube-rbac-proxy"
Jul 30 17:52:32.671778 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671652    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kserve-container"
Jul 30 17:52:32.671778 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.671659    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e86bd96-77d8-411c-9dbf-f01d951d57fc" containerName="kube-rbac-proxy"
Jul 30 17:52:32.676100 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.676080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.678503 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678480    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 30 17:52:32.678617 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678598    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:52:32.678675 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678598    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:52:32.678848 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678825    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:52:32.678956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:52:32.678956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678880    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:52:32.678956 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.678879    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 30 17:52:32.685852 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.685828    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"]
Jul 30 17:52:32.774484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.770360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.774484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.770393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.774484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.770418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.774484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.770514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64td\" (UniqueName: \"kubernetes.io/projected/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-api-access-p64td\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.774484 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.770552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.871531 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.871499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.871693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.871561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64td\" (UniqueName: \"kubernetes.io/projected/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-api-access-p64td\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.871693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.871600    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.871693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.871640    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.871693 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.871676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.872016 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.871996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.872250 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.872233    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.872250 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.872243    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.873899 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.873872    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.880326 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.880306    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64td\" (UniqueName: \"kubernetes.io/projected/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-api-access-p64td\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:32.988456 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:32.988427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:33.117309 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:33.117280    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"]
Jul 30 17:52:33.119551 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:52:33.119524    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7964728a_a3a7_4cb7_a3d4_3455dfc4d3eb.slice/crio-e97f7bbefdc6e2b56be1563a489cd22ca17bfc4f8d3cbc74de10e5189995a869 WatchSource:0}: Error finding container e97f7bbefdc6e2b56be1563a489cd22ca17bfc4f8d3cbc74de10e5189995a869: Status 404 returned error can't find the container with id e97f7bbefdc6e2b56be1563a489cd22ca17bfc4f8d3cbc74de10e5189995a869
Jul 30 17:52:33.486047 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:33.486011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerStarted","Data":"3b2ba9ceb6c5c97f95e2f02576651c6df974858f63b47e275b7c04035b04417b"}
Jul 30 17:52:33.486230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:33.486068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerStarted","Data":"e97f7bbefdc6e2b56be1563a489cd22ca17bfc4f8d3cbc74de10e5189995a869"}
Jul 30 17:52:37.502350 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:37.502311    2569 generic.go:356] "Generic (PLEG): container finished" podID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerID="3b2ba9ceb6c5c97f95e2f02576651c6df974858f63b47e275b7c04035b04417b" exitCode=0
Jul 30 17:52:37.502730 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:37.502385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerDied","Data":"3b2ba9ceb6c5c97f95e2f02576651c6df974858f63b47e275b7c04035b04417b"}
Jul 30 17:52:38.512480 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:38.512448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerStarted","Data":"43e9fef27e5c9f98fdcf58e7cf555d491c27824bac33cd89d9a289d763b68412"}
Jul 30 17:52:38.512930 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:38.512528    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerStarted","Data":"bd31306dbbac334a6e1557ee9266f46b08befe2503571eb4038c69e2d4f373ce"}
Jul 30 17:52:38.512930 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:38.512820    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:38.512930 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:38.512869    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:38.514208 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:38.514173    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:52:38.535197 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:38.535128    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podStartSLOduration=2.86263845 podStartE2EDuration="6.535116297s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.672477847s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:52:38.534623551 +0000 UTC m=+3126.567433887" watchObservedRunningTime="2026-07-30 17:52:38.535116297 +0000 UTC m=+3126.567926634"
Jul 30 17:52:39.516682 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:39.516642    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:52:44.522227 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:44.522200    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:52:44.522899 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:44.522732    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:52:54.522988 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:52:54.522898    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:53:04.523503 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:53:04.523452    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:53:14.522707 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:53:14.522665    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:53:24.523506 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:53:24.523460    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:53:34.523703 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:53:34.523659    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:53:44.522752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:53:44.522702    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:53:54.523465 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:53:54.523405    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:54:04.523355 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:04.523320    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:54:13.682659 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:13.682627    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"]
Jul 30 17:54:13.683120 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:13.682901    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" containerID="cri-o://bd31306dbbac334a6e1557ee9266f46b08befe2503571eb4038c69e2d4f373ce" gracePeriod=30
Jul 30 17:54:13.683120 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:13.682915    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kube-rbac-proxy" containerID="cri-o://43e9fef27e5c9f98fdcf58e7cf555d491c27824bac33cd89d9a289d763b68412" gracePeriod=30
Jul 30 17:54:13.866035 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:13.866004    2569 generic.go:356] "Generic (PLEG): container finished" podID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerID="43e9fef27e5c9f98fdcf58e7cf555d491c27824bac33cd89d9a289d763b68412" exitCode=2
Jul 30 17:54:13.866188 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:13.866070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerDied","Data":"43e9fef27e5c9f98fdcf58e7cf555d491c27824bac33cd89d9a289d763b68412"}
Jul 30 17:54:14.517057 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:14.517013    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 30 17:54:14.522698 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:14.522663    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 30 17:54:16.878809 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:16.878771    2569 generic.go:356] "Generic (PLEG): container finished" podID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerID="bd31306dbbac334a6e1557ee9266f46b08befe2503571eb4038c69e2d4f373ce" exitCode=0
Jul 30 17:54:16.879132 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:16.878827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerDied","Data":"bd31306dbbac334a6e1557ee9266f46b08befe2503571eb4038c69e2d4f373ce"}
Jul 30 17:54:16.918816 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:16.918789    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:54:17.013220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013125    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-proxy-tls\") pod \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") "
Jul 30 17:54:17.013220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kserve-provision-location\") pod \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") "
Jul 30 17:54:17.013220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013200    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p64td\" (UniqueName: \"kubernetes.io/projected/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-api-access-p64td\") pod \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") "
Jul 30 17:54:17.013529 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013235    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-cabundle-cert\") pod \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") "
Jul 30 17:54:17.013529 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013302    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-rbac-proxy-sar-config\") pod \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\" (UID: \"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb\") "
Jul 30 17:54:17.013641 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013562    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" (UID: "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:54:17.013696 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013641    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" (UID: "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:54:17.013774 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.013751    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" (UID: "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:54:17.015380 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.015360    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-api-access-p64td" (OuterVolumeSpecName: "kube-api-access-p64td") pod "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" (UID: "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb"). InnerVolumeSpecName "kube-api-access-p64td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:54:17.015458 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.015445    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" (UID: "7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:54:17.113880 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.113842    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:54:17.113880 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.113877    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:54:17.114093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.113892    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:54:17.114093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.113909    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64td\" (UniqueName: \"kubernetes.io/projected/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-kube-api-access-p64td\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:54:17.114093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.113920    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:54:17.884791 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.884757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6" event={"ID":"7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb","Type":"ContainerDied","Data":"e97f7bbefdc6e2b56be1563a489cd22ca17bfc4f8d3cbc74de10e5189995a869"}
Jul 30 17:54:17.885234 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.884829    2569 scope.go:117] "RemoveContainer" containerID="43e9fef27e5c9f98fdcf58e7cf555d491c27824bac33cd89d9a289d763b68412"
Jul 30 17:54:17.885234 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.884772    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"
Jul 30 17:54:17.893050 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.893031    2569 scope.go:117] "RemoveContainer" containerID="bd31306dbbac334a6e1557ee9266f46b08befe2503571eb4038c69e2d4f373ce"
Jul 30 17:54:17.900189 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.900168    2569 scope.go:117] "RemoveContainer" containerID="3b2ba9ceb6c5c97f95e2f02576651c6df974858f63b47e275b7c04035b04417b"
Jul 30 17:54:17.909865 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.909842    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"]
Jul 30 17:54:17.916609 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:17.916589    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-pcvz6"]
Jul 30 17:54:18.651776 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.651747    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" path="/var/lib/kubelet/pods/7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb/volumes"
Jul 30 17:54:18.879686 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.879646    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"]
Jul 30 17:54:18.879985 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.879968    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kube-rbac-proxy"
Jul 30 17:54:18.880081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.879989    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kube-rbac-proxy"
Jul 30 17:54:18.880081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.880015    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container"
Jul 30 17:54:18.880081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.880024    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container"
Jul 30 17:54:18.880081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.880035    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="storage-initializer"
Jul 30 17:54:18.880081 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.880044    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="storage-initializer"
Jul 30 17:54:18.880390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.880120    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kserve-container"
Jul 30 17:54:18.880390 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.880131    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7964728a-a3a7-4cb7-a3d4-3455dfc4d3eb" containerName="kube-rbac-proxy"
Jul 30 17:54:18.884941 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.884917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:18.887385 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.887363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b860f5-predictor-serving-cert\""
Jul 30 17:54:18.887560 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.887435    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:54:18.887645 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.887602    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:54:18.887703 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.887688    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:54:18.887885 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.887869    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b860f5-kube-rbac-proxy-sar-config\""
Jul 30 17:54:18.888509 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.888491    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:54:18.888509 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.888504    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:54:18.894613 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:18.894429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"]
Jul 30 17:54:19.028069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.028031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-cabundle-cert\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.028069 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.028070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kserve-provision-location\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.028366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.028124    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61740d-3c98-4336-a4f3-eefaf1380c7c-proxy-tls\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.028366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.028199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcfb\" (UniqueName: \"kubernetes.io/projected/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-api-access-cgcfb\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.028366 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.028216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129002 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.128969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcfb\" (UniqueName: \"kubernetes.io/projected/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-api-access-cgcfb\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129002 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129209 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-cabundle-cert\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129209 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kserve-provision-location\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129209 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129100    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61740d-3c98-4336-a4f3-eefaf1380c7c-proxy-tls\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129623 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129601    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kserve-provision-location\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129762 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-cabundle-cert\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.129888 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.129866    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.131529 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.131512    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61740d-3c98-4336-a4f3-eefaf1380c7c-proxy-tls\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.138752 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.138730    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcfb\" (UniqueName: \"kubernetes.io/projected/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-api-access-cgcfb\") pod \"isvc-primary-b860f5-predictor-7c5b56bc79-w7srr\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") " pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.199368 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.199338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:19.327174 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.327149    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"]
Jul 30 17:54:19.332183 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:54:19.332145    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a61740d_3c98_4336_a4f3_eefaf1380c7c.slice/crio-9002130ad3f5ec8d548553c80a786730daf858cf1427d87df8a394ca38fbd9a7 WatchSource:0}: Error finding container 9002130ad3f5ec8d548553c80a786730daf858cf1427d87df8a394ca38fbd9a7: Status 404 returned error can't find the container with id 9002130ad3f5ec8d548553c80a786730daf858cf1427d87df8a394ca38fbd9a7
Jul 30 17:54:19.894717 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.894684    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerStarted","Data":"33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e"}
Jul 30 17:54:19.895108 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:19.894745    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerStarted","Data":"9002130ad3f5ec8d548553c80a786730daf858cf1427d87df8a394ca38fbd9a7"}
Jul 30 17:54:23.910405 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:23.910374    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerID="33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e" exitCode=0
Jul 30 17:54:23.910795 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:23.910436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerDied","Data":"33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e"}
Jul 30 17:54:24.916827 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:24.916794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerStarted","Data":"7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93"}
Jul 30 17:54:24.917204 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:24.916853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerStarted","Data":"864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1"}
Jul 30 17:54:24.917204 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:24.917096    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:24.939025 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:24.938972    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podStartSLOduration=3.097063836 podStartE2EDuration="6.938956487s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841892651s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:54:24.937360741 +0000 UTC m=+3232.970171078" watchObservedRunningTime="2026-07-30 17:54:24.938956487 +0000 UTC m=+3232.971766823"
Jul 30 17:54:25.921335 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:25.921302    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:25.922543 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:25.922512    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:54:26.925202 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:26.925161    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:54:31.929972 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:31.929945    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:54:31.930417 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:31.930390    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:54:41.930462 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:41.930419    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:54:51.930441 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:54:51.930397    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:55:01.930898 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:55:01.930855    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:55:11.930508 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:55:11.930463    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:55:21.931372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:55:21.931335    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:55:31.930882 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:55:31.930841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 30 17:55:40.651815 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:55:40.651774    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:56:04.073743 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.073659    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"]
Jul 30 17:56:04.076892 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.076875    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.079215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.079187    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b860f5-kube-rbac-proxy-sar-config\""
Jul 30 17:56:04.079215 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.079193    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-b860f5\""
Jul 30 17:56:04.079450 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.079430    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-b860f5-predictor-serving-cert\""
Jul 30 17:56:04.079912 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.079898    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-b860f5-dockercfg-mdvz7\""
Jul 30 17:56:04.087138 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.087104    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"]
Jul 30 17:56:04.166803 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.166768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbpr\" (UniqueName: \"kubernetes.io/projected/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-api-access-hmbpr\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.166938 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.166820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.166938 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.166868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.166938 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.166909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a459a43-4ebb-4658-9b01-7793fe50214b-kserve-provision-location\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.166938 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.166936    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-cabundle-cert\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268023 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.267980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a459a43-4ebb-4658-9b01-7793fe50214b-kserve-provision-location\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268023 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-cabundle-cert\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbpr\" (UniqueName: \"kubernetes.io/projected/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-api-access-hmbpr\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268286 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268116    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268286 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:04.268250    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-serving-cert: secret "isvc-secondary-b860f5-predictor-serving-cert" not found
Jul 30 17:56:04.268469 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:04.268356    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls podName:7a459a43-4ebb-4658-9b01-7793fe50214b nodeName:}" failed. No retries permitted until 2026-07-30 17:56:04.768334333 +0000 UTC m=+3332.801144653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls") pod "isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" (UID: "7a459a43-4ebb-4658-9b01-7793fe50214b") : secret "isvc-secondary-b860f5-predictor-serving-cert" not found
Jul 30 17:56:04.268513 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268497    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a459a43-4ebb-4658-9b01-7793fe50214b-kserve-provision-location\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268775 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268757    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.268833 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.268817    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-cabundle-cert\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.277563 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.277541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbpr\" (UniqueName: \"kubernetes.io/projected/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-api-access-hmbpr\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.773089 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.773032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.775505 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.775474    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls\") pod \"isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") " pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:04.987554 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:04.987501    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:05.115572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:05.115455    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"]
Jul 30 17:56:05.118174 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:56:05.118145    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a459a43_4ebb_4658_9b01_7793fe50214b.slice/crio-550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105 WatchSource:0}: Error finding container 550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105: Status 404 returned error can't find the container with id 550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105
Jul 30 17:56:05.284665 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:05.284610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" event={"ID":"7a459a43-4ebb-4658-9b01-7793fe50214b","Type":"ContainerStarted","Data":"dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761"}
Jul 30 17:56:05.284867 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:05.284685    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" event={"ID":"7a459a43-4ebb-4658-9b01-7793fe50214b","Type":"ContainerStarted","Data":"550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105"}
Jul 30 17:56:10.304110 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:10.304080    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/0.log"
Jul 30 17:56:10.304555 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:10.304117    2569 generic.go:356] "Generic (PLEG): container finished" podID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerID="dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761" exitCode=1
Jul 30 17:56:10.304555 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:10.304170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" event={"ID":"7a459a43-4ebb-4658-9b01-7793fe50214b","Type":"ContainerDied","Data":"dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761"}
Jul 30 17:56:11.314728 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:11.314688    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/0.log"
Jul 30 17:56:11.315117 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:11.314795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" event={"ID":"7a459a43-4ebb-4658-9b01-7793fe50214b","Type":"ContainerStarted","Data":"97c37b4aefad270fb604d433d67cd1b30b7d59db11e35db2866c2ee82cb4e752"}
Jul 30 17:56:14.326597 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.326510    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/1.log"
Jul 30 17:56:14.327024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.326971    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/0.log"
Jul 30 17:56:14.327024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.327014    2569 generic.go:356] "Generic (PLEG): container finished" podID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerID="97c37b4aefad270fb604d433d67cd1b30b7d59db11e35db2866c2ee82cb4e752" exitCode=1
Jul 30 17:56:14.327163 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.327052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" event={"ID":"7a459a43-4ebb-4658-9b01-7793fe50214b","Type":"ContainerDied","Data":"97c37b4aefad270fb604d433d67cd1b30b7d59db11e35db2866c2ee82cb4e752"}
Jul 30 17:56:14.327163 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.327128    2569 scope.go:117] "RemoveContainer" containerID="dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761"
Jul 30 17:56:14.327540 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.327505    2569 scope.go:117] "RemoveContainer" containerID="dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761"
Jul 30 17:56:14.337644 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:14.337621    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_kserve-ci-e2e-test(7a459a43-4ebb-4658-9b01-7793fe50214b)\"" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b"
Jul 30 17:56:14.337907 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:14.337879    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_kserve-ci-e2e-test_7a459a43-4ebb-4658-9b01-7793fe50214b_0 in pod sandbox 550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105 from index: no such id: 'dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761'" containerID="dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761"
Jul 30 17:56:14.337963 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:14.337919    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_kserve-ci-e2e-test_7a459a43-4ebb-4658-9b01-7793fe50214b_0 in pod sandbox 550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105 from index: no such id: 'dc9f5795cc1f83831457796bed4713cbbd3fd8a180f75b8e5573457a815b2761'"
Jul 30 17:56:15.332991 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:15.332967    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/1.log"
Jul 30 17:56:22.083995 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.083954    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"]
Jul 30 17:56:22.084423 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.084230    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container" containerID="cri-o://864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1" gracePeriod=30
Jul 30 17:56:22.084423 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.084298    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kube-rbac-proxy" containerID="cri-o://7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93" gracePeriod=30
Jul 30 17:56:22.154893 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.154858    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"]
Jul 30 17:56:22.278206 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.278183    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/1.log"
Jul 30 17:56:22.280527 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.280512    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:22.317403 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317369    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-cabundle-cert\") pod \"7a459a43-4ebb-4658-9b01-7793fe50214b\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") "
Jul 30 17:56:22.317589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317417    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmbpr\" (UniqueName: \"kubernetes.io/projected/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-api-access-hmbpr\") pod \"7a459a43-4ebb-4658-9b01-7793fe50214b\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") "
Jul 30 17:56:22.317589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317456    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-rbac-proxy-sar-config\") pod \"7a459a43-4ebb-4658-9b01-7793fe50214b\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") "
Jul 30 17:56:22.317589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317490    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls\") pod \"7a459a43-4ebb-4658-9b01-7793fe50214b\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") "
Jul 30 17:56:22.317589 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317539    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a459a43-4ebb-4658-9b01-7793fe50214b-kserve-provision-location\") pod \"7a459a43-4ebb-4658-9b01-7793fe50214b\" (UID: \"7a459a43-4ebb-4658-9b01-7793fe50214b\") "
Jul 30 17:56:22.317839 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317814    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a459a43-4ebb-4658-9b01-7793fe50214b" (UID: "7a459a43-4ebb-4658-9b01-7793fe50214b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:56:22.317908 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317820    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a459a43-4ebb-4658-9b01-7793fe50214b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a459a43-4ebb-4658-9b01-7793fe50214b" (UID: "7a459a43-4ebb-4658-9b01-7793fe50214b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:56:22.318010 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.317898    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a459a43-4ebb-4658-9b01-7793fe50214b" (UID: "7a459a43-4ebb-4658-9b01-7793fe50214b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:56:22.319624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.319594    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a459a43-4ebb-4658-9b01-7793fe50214b" (UID: "7a459a43-4ebb-4658-9b01-7793fe50214b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:56:22.319624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.319616    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-api-access-hmbpr" (OuterVolumeSpecName: "kube-api-access-hmbpr") pod "7a459a43-4ebb-4658-9b01-7793fe50214b" (UID: "7a459a43-4ebb-4658-9b01-7793fe50214b"). InnerVolumeSpecName "kube-api-access-hmbpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:56:22.366486 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.366396    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerID="7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93" exitCode=2
Jul 30 17:56:22.366630 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.366474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerDied","Data":"7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93"}
Jul 30 17:56:22.367621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.367603    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t_7a459a43-4ebb-4658-9b01-7793fe50214b/storage-initializer/1.log"
Jul 30 17:56:22.367744 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.367663    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t" event={"ID":"7a459a43-4ebb-4658-9b01-7793fe50214b","Type":"ContainerDied","Data":"550a2f3e6e7cc96cc8abf717d1c80071844f576e20bf00f1703adae2fb3c3105"}
Jul 30 17:56:22.367744 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.367707    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"
Jul 30 17:56:22.367873 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.367713    2569 scope.go:117] "RemoveContainer" containerID="97c37b4aefad270fb604d433d67cd1b30b7d59db11e35db2866c2ee82cb4e752"
Jul 30 17:56:22.412347 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.412319    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"]
Jul 30 17:56:22.417613 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.417589    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-b860f5-predictor-fc95bfb56-t4k8t"]
Jul 30 17:56:22.418516 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.418496    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a459a43-4ebb-4658-9b01-7793fe50214b-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:22.418607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.418522    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:22.418607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.418538    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbpr\" (UniqueName: \"kubernetes.io/projected/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-api-access-hmbpr\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:22.418607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.418551    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a459a43-4ebb-4658-9b01-7793fe50214b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:22.418607 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.418563    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a459a43-4ebb-4658-9b01-7793fe50214b-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:22.652473 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:22.652397    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" path="/var/lib/kubelet/pods/7a459a43-4ebb-4658-9b01-7793fe50214b/volumes"
Jul 30 17:56:26.724761 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.724737    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:56:26.752795 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.752763    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kserve-provision-location\") pod \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") "
Jul 30 17:56:26.752962 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.752823    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-cabundle-cert\") pod \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") "
Jul 30 17:56:26.753029 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.752957    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgcfb\" (UniqueName: \"kubernetes.io/projected/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-api-access-cgcfb\") pod \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") "
Jul 30 17:56:26.753029 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753006    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61740d-3c98-4336-a4f3-eefaf1380c7c-proxy-tls\") pod \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") "
Jul 30 17:56:26.753151 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753077    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-rbac-proxy-sar-config\") pod \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\" (UID: \"6a61740d-3c98-4336-a4f3-eefaf1380c7c\") "
Jul 30 17:56:26.753151 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753118    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a61740d-3c98-4336-a4f3-eefaf1380c7c" (UID: "6a61740d-3c98-4336-a4f3-eefaf1380c7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:56:26.753282 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753169    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a61740d-3c98-4336-a4f3-eefaf1380c7c" (UID: "6a61740d-3c98-4336-a4f3-eefaf1380c7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:56:26.753491 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753463    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a61740d-3c98-4336-a4f3-eefaf1380c7c" (UID: "6a61740d-3c98-4336-a4f3-eefaf1380c7c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:56:26.753623 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753498    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:26.753623 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.753521    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:26.755066 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.755043    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-api-access-cgcfb" (OuterVolumeSpecName: "kube-api-access-cgcfb") pod "6a61740d-3c98-4336-a4f3-eefaf1380c7c" (UID: "6a61740d-3c98-4336-a4f3-eefaf1380c7c"). InnerVolumeSpecName "kube-api-access-cgcfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:56:26.755170 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.755134    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a61740d-3c98-4336-a4f3-eefaf1380c7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a61740d-3c98-4336-a4f3-eefaf1380c7c" (UID: "6a61740d-3c98-4336-a4f3-eefaf1380c7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:56:26.854860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.854771    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a61740d-3c98-4336-a4f3-eefaf1380c7c-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:26.854860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.854804    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:26.854860 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:26.854817    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcfb\" (UniqueName: \"kubernetes.io/projected/6a61740d-3c98-4336-a4f3-eefaf1380c7c-kube-api-access-cgcfb\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:27.321588 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321550    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"]
Jul 30 17:56:27.321865 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321850    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kube-rbac-proxy"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321867    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kube-rbac-proxy"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321877    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerName="storage-initializer"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321882    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerName="storage-initializer"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321896    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321903    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321909    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="storage-initializer"
Jul 30 17:56:27.321913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321914    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="storage-initializer"
Jul 30 17:56:27.322155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321957    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kserve-container"
Jul 30 17:56:27.322155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321964    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerName="storage-initializer"
Jul 30 17:56:27.322155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.321972    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerName="kube-rbac-proxy"
Jul 30 17:56:27.322155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.322016    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerName="storage-initializer"
Jul 30 17:56:27.322155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.322022    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerName="storage-initializer"
Jul 30 17:56:27.322155 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.322064    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a459a43-4ebb-4658-9b01-7793fe50214b" containerName="storage-initializer"
Jul 30 17:56:27.325222 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.325204    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.327850 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.327780    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f2962e-predictor-serving-cert\""
Jul 30 17:56:27.328113 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.328093    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f2962e-dockercfg-hzhpp\""
Jul 30 17:56:27.328239 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.328144    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f2962e-kube-rbac-proxy-sar-config\""
Jul 30 17:56:27.328239 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.328169    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f2962e\""
Jul 30 17:56:27.338291 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.338249    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"]
Jul 30 17:56:27.358700 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.358670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-cabundle-cert\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.358817 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.358709    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.358817 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.358791    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.358913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.358835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4wrh\" (UniqueName: \"kubernetes.io/projected/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-api-access-k4wrh\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.358913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.358853    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kserve-provision-location\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.388024 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.387997    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" containerID="864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1" exitCode=0
Jul 30 17:56:27.388133 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.388052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerDied","Data":"864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1"}
Jul 30 17:56:27.388133 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.388063    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"
Jul 30 17:56:27.388133 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.388099    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr" event={"ID":"6a61740d-3c98-4336-a4f3-eefaf1380c7c","Type":"ContainerDied","Data":"9002130ad3f5ec8d548553c80a786730daf858cf1427d87df8a394ca38fbd9a7"}
Jul 30 17:56:27.388133 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.388127    2569 scope.go:117] "RemoveContainer" containerID="7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93"
Jul 30 17:56:27.396271 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.396236    2569 scope.go:117] "RemoveContainer" containerID="864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1"
Jul 30 17:56:27.403231 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.403212    2569 scope.go:117] "RemoveContainer" containerID="33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e"
Jul 30 17:56:27.410614 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.410594    2569 scope.go:117] "RemoveContainer" containerID="7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93"
Jul 30 17:56:27.410844 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:27.410825    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93\": container with ID starting with 7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93 not found: ID does not exist" containerID="7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93"
Jul 30 17:56:27.410913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.410851    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93"} err="failed to get container status \"7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93\": rpc error: code = NotFound desc = could not find container \"7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93\": container with ID starting with 7b6c0a4e6f9c997732b7c91f6eebf1472a01794b238388a08f5b00f2537d2e93 not found: ID does not exist"
Jul 30 17:56:27.410913 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.410867    2569 scope.go:117] "RemoveContainer" containerID="864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1"
Jul 30 17:56:27.411103 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:27.411086    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1\": container with ID starting with 864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1 not found: ID does not exist" containerID="864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1"
Jul 30 17:56:27.411149 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.411106    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1"} err="failed to get container status \"864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1\": rpc error: code = NotFound desc = could not find container \"864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1\": container with ID starting with 864e3efa8dc5a53bfbf7280e4606818c5b3d08b4fc7401e0029e0e9641529ab1 not found: ID does not exist"
Jul 30 17:56:27.411149 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.411119    2569 scope.go:117] "RemoveContainer" containerID="33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e"
Jul 30 17:56:27.411230 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.411152    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"]
Jul 30 17:56:27.411322 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:27.411307    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e\": container with ID starting with 33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e not found: ID does not exist" containerID="33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e"
Jul 30 17:56:27.411372 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.411325    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e"} err="failed to get container status \"33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e\": rpc error: code = NotFound desc = could not find container \"33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e\": container with ID starting with 33e2bb3f44be7a94df4187f862c9849fa7247257730c4907e74d0ce7722b375e not found: ID does not exist"
Jul 30 17:56:27.415236 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.415217    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b860f5-predictor-7c5b56bc79-w7srr"]
Jul 30 17:56:27.459477 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.459449    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-cabundle-cert\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.459572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.459483    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.459572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.459515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.459572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.459547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4wrh\" (UniqueName: \"kubernetes.io/projected/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-api-access-k4wrh\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.459572 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.459564    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kserve-provision-location\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.459771 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:27.459681    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-serving-cert: secret "isvc-init-fail-f2962e-predictor-serving-cert" not found
Jul 30 17:56:27.459771 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:27.459750    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls podName:afc92c4e-0df1-4721-b48d-5cb1bf9e732b nodeName:}" failed. No retries permitted until 2026-07-30 17:56:27.959729342 +0000 UTC m=+3355.992539657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls") pod "isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" (UID: "afc92c4e-0df1-4721-b48d-5cb1bf9e732b") : secret "isvc-init-fail-f2962e-predictor-serving-cert" not found
Jul 30 17:56:27.459922 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.459907    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kserve-provision-location\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.460146 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.460130    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.460191 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.460155    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-cabundle-cert\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.471898 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.471868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4wrh\" (UniqueName: \"kubernetes.io/projected/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-api-access-k4wrh\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.963456 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.963418    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:27.965933 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:27.965906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls\") pod \"isvc-init-fail-f2962e-predictor-d56689d86-s9ntm\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") " pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:28.235582 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:28.235493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:28.367005 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:28.366961    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"]
Jul 30 17:56:28.369579 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:56:28.369548    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc92c4e_0df1_4721_b48d_5cb1bf9e732b.slice/crio-8c5e47e9a47176f2c46924740d6573f3772b9e77b479e7935fa2027709ef2aed WatchSource:0}: Error finding container 8c5e47e9a47176f2c46924740d6573f3772b9e77b479e7935fa2027709ef2aed: Status 404 returned error can't find the container with id 8c5e47e9a47176f2c46924740d6573f3772b9e77b479e7935fa2027709ef2aed
Jul 30 17:56:28.393667 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:28.393641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" event={"ID":"afc92c4e-0df1-4721-b48d-5cb1bf9e732b","Type":"ContainerStarted","Data":"8c5e47e9a47176f2c46924740d6573f3772b9e77b479e7935fa2027709ef2aed"}
Jul 30 17:56:28.651726 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:28.651627    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a61740d-3c98-4336-a4f3-eefaf1380c7c" path="/var/lib/kubelet/pods/6a61740d-3c98-4336-a4f3-eefaf1380c7c/volumes"
Jul 30 17:56:29.398629 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:29.398588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" event={"ID":"afc92c4e-0df1-4721-b48d-5cb1bf9e732b","Type":"ContainerStarted","Data":"4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27"}
Jul 30 17:56:31.407059 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:31.406980    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f2962e-predictor-d56689d86-s9ntm_afc92c4e-0df1-4721-b48d-5cb1bf9e732b/storage-initializer/0.log"
Jul 30 17:56:31.407059 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:31.407019    2569 generic.go:356] "Generic (PLEG): container finished" podID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerID="4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27" exitCode=1
Jul 30 17:56:31.407059 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:31.407057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" event={"ID":"afc92c4e-0df1-4721-b48d-5cb1bf9e732b","Type":"ContainerDied","Data":"4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27"}
Jul 30 17:56:32.326360 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:32.326325    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"]
Jul 30 17:56:32.412783 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:32.412756    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f2962e-predictor-d56689d86-s9ntm_afc92c4e-0df1-4721-b48d-5cb1bf9e732b/storage-initializer/0.log"
Jul 30 17:56:32.413221 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:32.412828    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" event={"ID":"afc92c4e-0df1-4721-b48d-5cb1bf9e732b","Type":"ContainerStarted","Data":"13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89"}
Jul 30 17:56:32.413221 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:32.412966    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer" containerID="cri-o://13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89" gracePeriod=30
Jul 30 17:56:36.360917 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.360894    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f2962e-predictor-d56689d86-s9ntm_afc92c4e-0df1-4721-b48d-5cb1bf9e732b/storage-initializer/1.log"
Jul 30 17:56:36.361332 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.361316    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f2962e-predictor-d56689d86-s9ntm_afc92c4e-0df1-4721-b48d-5cb1bf9e732b/storage-initializer/0.log"
Jul 30 17:56:36.363834 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.363818    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:36.427095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427023    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls\") pod \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") "
Jul 30 17:56:36.427095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427074    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-cabundle-cert\") pod \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") "
Jul 30 17:56:36.427296 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-rbac-proxy-sar-config\") pod \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") "
Jul 30 17:56:36.427296 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427163    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4wrh\" (UniqueName: \"kubernetes.io/projected/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-api-access-k4wrh\") pod \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") "
Jul 30 17:56:36.427296 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427204    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kserve-provision-location\") pod \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\" (UID: \"afc92c4e-0df1-4721-b48d-5cb1bf9e732b\") "
Jul 30 17:56:36.427511 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427485    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afc92c4e-0df1-4721-b48d-5cb1bf9e732b" (UID: "afc92c4e-0df1-4721-b48d-5cb1bf9e732b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:56:36.427694 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427608    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afc92c4e-0df1-4721-b48d-5cb1bf9e732b" (UID: "afc92c4e-0df1-4721-b48d-5cb1bf9e732b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:56:36.427798 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427657    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afc92c4e-0df1-4721-b48d-5cb1bf9e732b" (UID: "afc92c4e-0df1-4721-b48d-5cb1bf9e732b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:56:36.427798 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.427739    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f2962e-predictor-d56689d86-s9ntm_afc92c4e-0df1-4721-b48d-5cb1bf9e732b/storage-initializer/1.log"
Jul 30 17:56:36.428220 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.428199    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f2962e-predictor-d56689d86-s9ntm_afc92c4e-0df1-4721-b48d-5cb1bf9e732b/storage-initializer/0.log"
Jul 30 17:56:36.428343 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.428248    2569 generic.go:356] "Generic (PLEG): container finished" podID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerID="13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89" exitCode=1
Jul 30 17:56:36.428343 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.428310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" event={"ID":"afc92c4e-0df1-4721-b48d-5cb1bf9e732b","Type":"ContainerDied","Data":"13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89"}
Jul 30 17:56:36.428452 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.428338    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"
Jul 30 17:56:36.428452 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.428374    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm" event={"ID":"afc92c4e-0df1-4721-b48d-5cb1bf9e732b","Type":"ContainerDied","Data":"8c5e47e9a47176f2c46924740d6573f3772b9e77b479e7935fa2027709ef2aed"}
Jul 30 17:56:36.428452 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.428412    2569 scope.go:117] "RemoveContainer" containerID="13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89"
Jul 30 17:56:36.429338 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.429316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afc92c4e-0df1-4721-b48d-5cb1bf9e732b" (UID: "afc92c4e-0df1-4721-b48d-5cb1bf9e732b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:56:36.429440 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.429401    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-api-access-k4wrh" (OuterVolumeSpecName: "kube-api-access-k4wrh") pod "afc92c4e-0df1-4721-b48d-5cb1bf9e732b" (UID: "afc92c4e-0df1-4721-b48d-5cb1bf9e732b"). InnerVolumeSpecName "kube-api-access-k4wrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:56:36.441045 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.441024    2569 scope.go:117] "RemoveContainer" containerID="4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27"
Jul 30 17:56:36.447707 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.447689    2569 scope.go:117] "RemoveContainer" containerID="13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89"
Jul 30 17:56:36.447935 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:36.447916    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89\": container with ID starting with 13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89 not found: ID does not exist" containerID="13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89"
Jul 30 17:56:36.448001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.447947    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89"} err="failed to get container status \"13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89\": rpc error: code = NotFound desc = could not find container \"13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89\": container with ID starting with 13299a6f2eadc62b83efcb3a486dffca7d7ba4e67dd1beee367792aec5328b89 not found: ID does not exist"
Jul 30 17:56:36.448001 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.447973    2569 scope.go:117] "RemoveContainer" containerID="4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27"
Jul 30 17:56:36.448232 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:56:36.448213    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27\": container with ID starting with 4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27 not found: ID does not exist" containerID="4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27"
Jul 30 17:56:36.448344 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.448238    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27"} err="failed to get container status \"4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27\": rpc error: code = NotFound desc = could not find container \"4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27\": container with ID starting with 4a7f4177191b9907238334c148a24d1b21a85a387ab22d6151b70ca87483fa27 not found: ID does not exist"
Jul 30 17:56:36.528367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.528333    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4wrh\" (UniqueName: \"kubernetes.io/projected/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-api-access-k4wrh\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:36.528367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.528360    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:36.528367 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.528370    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:36.528580 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.528382    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:36.528580 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.528390    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afc92c4e-0df1-4721-b48d-5cb1bf9e732b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:56:36.764462 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.764431    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"]
Jul 30 17:56:36.768363 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:36.768337    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f2962e-predictor-d56689d86-s9ntm"]
Jul 30 17:56:37.554754 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.554717    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"]
Jul 30 17:56:37.555095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.555012    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer"
Jul 30 17:56:37.555095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.555025    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer"
Jul 30 17:56:37.555095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.555078    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer"
Jul 30 17:56:37.555095 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.555088    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer"
Jul 30 17:56:37.555236 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.555149    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer"
Jul 30 17:56:37.555236 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.555159    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" containerName="storage-initializer"
Jul 30 17:56:37.559347 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.559326    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.561944 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.561915    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 17:56:37.562858 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.562838    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 17:56:37.563111 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.563093    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 30 17:56:37.563111 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.563107    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 30 17:56:37.563302 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.563239    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 17:56:37.563302 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.563293    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 17:56:37.563420 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.563329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 17:56:37.570633 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.570611    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"]
Jul 30 17:56:37.636467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.636431    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.636467 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.636470    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.636650 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.636496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.636650 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.636566    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl8s\" (UniqueName: \"kubernetes.io/projected/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-api-access-ddl8s\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.636650 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.636642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737037 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737095    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737130    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737195 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl8s\" (UniqueName: \"kubernetes.io/projected/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-api-access-ddl8s\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737516 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737489    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.737919 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.737897    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.738122 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.738103    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.739688 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.739670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.749815 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.749783    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl8s\" (UniqueName: \"kubernetes.io/projected/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-api-access-ddl8s\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-mltwp\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:37.869520 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:37.869429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:56:38.000203 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:38.000118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"]
Jul 30 17:56:38.002849 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:56:38.002819    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e0c308_aaa8_49ad_95c8_1d6f8b0da332.slice/crio-7362eddc4cf9064e0db9ed0680751072b34718691a93f014c90191f234abea8f WatchSource:0}: Error finding container 7362eddc4cf9064e0db9ed0680751072b34718691a93f014c90191f234abea8f: Status 404 returned error can't find the container with id 7362eddc4cf9064e0db9ed0680751072b34718691a93f014c90191f234abea8f
Jul 30 17:56:38.439025 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:38.438982    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerStarted","Data":"ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29"}
Jul 30 17:56:38.439228 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:38.439047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerStarted","Data":"7362eddc4cf9064e0db9ed0680751072b34718691a93f014c90191f234abea8f"}
Jul 30 17:56:38.652546 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:38.652514    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc92c4e-0df1-4721-b48d-5cb1bf9e732b" path="/var/lib/kubelet/pods/afc92c4e-0df1-4721-b48d-5cb1bf9e732b/volumes"
Jul 30 17:56:42.454624 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:42.454590    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerID="ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29" exitCode=0
Jul 30 17:56:42.455038 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:42.454646    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerDied","Data":"ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29"}
Jul 30 17:56:42.455765 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:56:42.455745    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 17:57:03.559021 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:03.558992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerStarted","Data":"0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1"}
Jul 30 17:57:04.565348 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:04.565311    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerStarted","Data":"d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8"}
Jul 30 17:57:04.565773 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:04.565430    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:57:04.587279 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:04.587204    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podStartSLOduration=2.884755422 podStartE2EDuration="27.587188916s" totalImagesPullingTime="20.90253093s" totalInitContainerRuntime="3.799902564s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:57:04.585577567 +0000 UTC m=+3392.618387905" watchObservedRunningTime="2026-07-30 17:57:04.587188916 +0000 UTC m=+3392.619999252"
Jul 30 17:57:05.569200 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:05.569167    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:57:05.570392 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:05.570365    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:57:06.572605 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:06.572557    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:57:11.576824 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:11.576790    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:57:11.577421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:11.577346    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:57:21.577659 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:21.577615    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:57:31.577808 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:31.577769    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:57:41.577877 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:41.577836    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:57:51.578059 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:57:51.578017    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:58:01.577408 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:01.577367    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:58:11.577621 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:11.577573    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:58:21.578357 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:21.578325    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:58:27.696973 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:27.696938    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"]
Jul 30 17:58:27.697460 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:27.697246    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" containerID="cri-o://0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1" gracePeriod=30
Jul 30 17:58:27.697460 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:27.697328    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kube-rbac-proxy" containerID="cri-o://d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8" gracePeriod=30
Jul 30 17:58:27.880093 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:27.880063    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerID="d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8" exitCode=2
Jul 30 17:58:27.880282 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:27.880141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerDied","Data":"d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8"}
Jul 30 17:58:31.573361 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:31.573322    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 30 17:58:31.577635 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:31.577613    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 30 17:58:32.330468 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.330443    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:58:32.459540 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459506    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kserve-provision-location\") pod \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") "
Jul 30 17:58:32.459739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-rbac-proxy-sar-config\") pod \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") "
Jul 30 17:58:32.459739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459641    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddl8s\" (UniqueName: \"kubernetes.io/projected/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-api-access-ddl8s\") pod \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") "
Jul 30 17:58:32.459739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459674    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-cabundle-cert\") pod \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") "
Jul 30 17:58:32.459739 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459712    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-proxy-tls\") pod \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\" (UID: \"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332\") "
Jul 30 17:58:32.459931 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459800    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" (UID: "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 17:58:32.459991 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459965    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" (UID: "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:58:32.460028 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.459979    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:58:32.460065 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.460028    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" (UID: "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 17:58:32.461810 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.461788    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" (UID: "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 17:58:32.461897 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.461853    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-api-access-ddl8s" (OuterVolumeSpecName: "kube-api-access-ddl8s") pod "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" (UID: "a9e0c308-aaa8-49ad-95c8-1d6f8b0da332"). InnerVolumeSpecName "kube-api-access-ddl8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 17:58:32.560585 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.560538    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddl8s\" (UniqueName: \"kubernetes.io/projected/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-api-access-ddl8s\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:58:32.560585 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.560581    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:58:32.560585 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.560592    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:58:32.560585 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.560600    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 17:58:32.847387 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847304    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"]
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847561    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kube-rbac-proxy"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847572    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kube-rbac-proxy"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847581    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847587    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847604    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="storage-initializer"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847610    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="storage-initializer"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847659    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kube-rbac-proxy"
Jul 30 17:58:32.847751 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.847667    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerName="kserve-container"
Jul 30 17:58:32.850108 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.850090    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:32.852584 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.852557    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 30 17:58:32.852703 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.852592    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 30 17:58:32.861128 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.861105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"]
Jul 30 17:58:32.899982 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.899956    2569 generic.go:356] "Generic (PLEG): container finished" podID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" containerID="0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1" exitCode=0
Jul 30 17:58:32.900144 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.900024    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"
Jul 30 17:58:32.900144 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.900047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerDied","Data":"0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1"}
Jul 30 17:58:32.900144 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.900107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp" event={"ID":"a9e0c308-aaa8-49ad-95c8-1d6f8b0da332","Type":"ContainerDied","Data":"7362eddc4cf9064e0db9ed0680751072b34718691a93f014c90191f234abea8f"}
Jul 30 17:58:32.900144 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.900134    2569 scope.go:117] "RemoveContainer" containerID="d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8"
Jul 30 17:58:32.907684 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.907662    2569 scope.go:117] "RemoveContainer" containerID="0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1"
Jul 30 17:58:32.914500 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.914483    2569 scope.go:117] "RemoveContainer" containerID="ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29"
Jul 30 17:58:32.920399 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.920379    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"]
Jul 30 17:58:32.921439 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.921422    2569 scope.go:117] "RemoveContainer" containerID="d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8"
Jul 30 17:58:32.921705 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:58:32.921673    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8\": container with ID starting with d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8 not found: ID does not exist" containerID="d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8"
Jul 30 17:58:32.921750 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.921707    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8"} err="failed to get container status \"d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8\": rpc error: code = NotFound desc = could not find container \"d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8\": container with ID starting with d8ec34830777938addc2ee2e9db7de4ecc3b4b9259c4ec7d759d014a010d82f8 not found: ID does not exist"
Jul 30 17:58:32.921750 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.921724    2569 scope.go:117] "RemoveContainer" containerID="0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1"
Jul 30 17:58:32.921941 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:58:32.921925    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1\": container with ID starting with 0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1 not found: ID does not exist" containerID="0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1"
Jul 30 17:58:32.921990 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.921946    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1"} err="failed to get container status \"0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1\": rpc error: code = NotFound desc = could not find container \"0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1\": container with ID starting with 0484a523f7310d5d16642fca7433503ec295efdd5e777ab890599e3c3e0373d1 not found: ID does not exist"
Jul 30 17:58:32.921990 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.921958    2569 scope.go:117] "RemoveContainer" containerID="ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29"
Jul 30 17:58:32.922141 ip-10-0-142-31 kubenswrapper[2569]: E0730 17:58:32.922124    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29\": container with ID starting with ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29 not found: ID does not exist" containerID="ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29"
Jul 30 17:58:32.922180 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.922143    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29"} err="failed to get container status \"ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29\": rpc error: code = NotFound desc = could not find container \"ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29\": container with ID starting with ad80779cbb55597a338ec9ccd5d5766745aeeb80610d3605a70fb2e0c41f2c29 not found: ID does not exist"
Jul 30 17:58:32.925637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.925617    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-mltwp"]
Jul 30 17:58:32.965421 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.965395    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:32.965504 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.965428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdq59\" (UniqueName: \"kubernetes.io/projected/daf96f80-b682-412f-b46d-51b2ff803a13-kube-api-access-rdq59\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:32.965504 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.965455    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf96f80-b682-412f-b46d-51b2ff803a13-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:32.965504 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.965497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf96f80-b682-412f-b46d-51b2ff803a13-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:32.965610 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:32.965556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.066601 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.066567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.066735 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.066604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdq59\" (UniqueName: \"kubernetes.io/projected/daf96f80-b682-412f-b46d-51b2ff803a13-kube-api-access-rdq59\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.066735 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.066627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf96f80-b682-412f-b46d-51b2ff803a13-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.066735 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.066653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf96f80-b682-412f-b46d-51b2ff803a13-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.066735 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.066699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.067092 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.067071    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf96f80-b682-412f-b46d-51b2ff803a13-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.067362 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.067345    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.067431 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.067409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.069087 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.069070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf96f80-b682-412f-b46d-51b2ff803a13-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.075942 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.075922    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdq59\" (UniqueName: \"kubernetes.io/projected/daf96f80-b682-412f-b46d-51b2ff803a13-kube-api-access-rdq59\") pod \"isvc-predictive-xgboost-predictor-695f978978-8g2xm\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.160153 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.160079    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:33.285416 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.285381    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"]
Jul 30 17:58:33.288206 ip-10-0-142-31 kubenswrapper[2569]: W0730 17:58:33.288178    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf96f80_b682_412f_b46d_51b2ff803a13.slice/crio-d9efd32487adc82765e366abb2bd6eb27e4d51b1cbe43c19d3d9e5b1cc5b75bb WatchSource:0}: Error finding container d9efd32487adc82765e366abb2bd6eb27e4d51b1cbe43c19d3d9e5b1cc5b75bb: Status 404 returned error can't find the container with id d9efd32487adc82765e366abb2bd6eb27e4d51b1cbe43c19d3d9e5b1cc5b75bb
Jul 30 17:58:33.905049 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.905009    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerStarted","Data":"a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab"}
Jul 30 17:58:33.905534 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:33.905083    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerStarted","Data":"d9efd32487adc82765e366abb2bd6eb27e4d51b1cbe43c19d3d9e5b1cc5b75bb"}
Jul 30 17:58:34.651669 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:34.651634    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e0c308-aaa8-49ad-95c8-1d6f8b0da332" path="/var/lib/kubelet/pods/a9e0c308-aaa8-49ad-95c8-1d6f8b0da332/volumes"
Jul 30 17:58:36.918574 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:36.918545    2569 generic.go:356] "Generic (PLEG): container finished" podID="daf96f80-b682-412f-b46d-51b2ff803a13" containerID="a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab" exitCode=0
Jul 30 17:58:36.918904 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:36.918617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerDied","Data":"a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab"}
Jul 30 17:58:37.924707 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:37.924667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerStarted","Data":"b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e"}
Jul 30 17:58:37.925191 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:37.924733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerStarted","Data":"02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313"}
Jul 30 17:58:37.925191 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:37.925014    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:37.925191 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:37.925141    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:37.926637 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:37.926610    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:58:37.948740 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:37.948691    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podStartSLOduration=2.414978741 podStartE2EDuration="5.948677608s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.533698867s" isStatefulPod=false podCreationTimestamp="2026-07-30 17:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 17:58:37.947481913 +0000 UTC m=+3485.980292273" watchObservedRunningTime="2026-07-30 17:58:37.948677608 +0000 UTC m=+3485.981488004"
Jul 30 17:58:38.928264 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:38.928224    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:58:43.932398 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:43.932364    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 17:58:43.933043 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:43.933014    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:58:53.933501 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:58:53.933414    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:59:03.933509 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:59:03.933465    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:59:13.933323 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:59:13.933279    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:59:23.933235 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:59:23.933192    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:59:33.932903 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:59:33.932851    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:59:43.933567 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:59:43.933523    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 17:59:53.934005 ip-10-0-142-31 kubenswrapper[2569]: I0730 17:59:53.933971    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 18:00:03.001716 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.001676    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"]
Jul 30 18:00:03.002200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.001971    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" containerID="cri-o://02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313" gracePeriod=30
Jul 30 18:00:03.002200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.002020    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kube-rbac-proxy" containerID="cri-o://b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e" gracePeriod=30
Jul 30 18:00:03.238557 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.238523    2569 generic.go:356] "Generic (PLEG): container finished" podID="daf96f80-b682-412f-b46d-51b2ff803a13" containerID="b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e" exitCode=2
Jul 30 18:00:03.238731 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.238605    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerDied","Data":"b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e"}
Jul 30 18:00:03.928529 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.928487    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 30 18:00:03.933831 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:03.933806    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 30 18:00:07.740139 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.740113    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 18:00:07.800658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.800550    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf96f80-b682-412f-b46d-51b2ff803a13-proxy-tls\") pod \"daf96f80-b682-412f-b46d-51b2ff803a13\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") "
Jul 30 18:00:07.800658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.800615    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf96f80-b682-412f-b46d-51b2ff803a13-kserve-provision-location\") pod \"daf96f80-b682-412f-b46d-51b2ff803a13\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") "
Jul 30 18:00:07.800658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.800654    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-kube-rbac-proxy-sar-config\") pod \"daf96f80-b682-412f-b46d-51b2ff803a13\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") "
Jul 30 18:00:07.800945 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.800674    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-cabundle-cert\") pod \"daf96f80-b682-412f-b46d-51b2ff803a13\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") "
Jul 30 18:00:07.800945 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.800696    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdq59\" (UniqueName: \"kubernetes.io/projected/daf96f80-b682-412f-b46d-51b2ff803a13-kube-api-access-rdq59\") pod \"daf96f80-b682-412f-b46d-51b2ff803a13\" (UID: \"daf96f80-b682-412f-b46d-51b2ff803a13\") "
Jul 30 18:00:07.801058 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.800977    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf96f80-b682-412f-b46d-51b2ff803a13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daf96f80-b682-412f-b46d-51b2ff803a13" (UID: "daf96f80-b682-412f-b46d-51b2ff803a13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:00:07.801058 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.801026    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daf96f80-b682-412f-b46d-51b2ff803a13" (UID: "daf96f80-b682-412f-b46d-51b2ff803a13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:00:07.801134 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.801050    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "daf96f80-b682-412f-b46d-51b2ff803a13" (UID: "daf96f80-b682-412f-b46d-51b2ff803a13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:00:07.802856 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.802830    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf96f80-b682-412f-b46d-51b2ff803a13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daf96f80-b682-412f-b46d-51b2ff803a13" (UID: "daf96f80-b682-412f-b46d-51b2ff803a13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:00:07.802940 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.802871    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf96f80-b682-412f-b46d-51b2ff803a13-kube-api-access-rdq59" (OuterVolumeSpecName: "kube-api-access-rdq59") pod "daf96f80-b682-412f-b46d-51b2ff803a13" (UID: "daf96f80-b682-412f-b46d-51b2ff803a13"). InnerVolumeSpecName "kube-api-access-rdq59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:00:07.901432 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.901379    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daf96f80-b682-412f-b46d-51b2ff803a13-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:00:07.901432 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.901427    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:00:07.901432 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.901441    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daf96f80-b682-412f-b46d-51b2ff803a13-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:00:07.901678 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.901454    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdq59\" (UniqueName: \"kubernetes.io/projected/daf96f80-b682-412f-b46d-51b2ff803a13-kube-api-access-rdq59\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:00:07.901678 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:07.901467    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf96f80-b682-412f-b46d-51b2ff803a13-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:00:08.191298 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191250    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"]
Jul 30 18:00:08.191538 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191527    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kube-rbac-proxy"
Jul 30 18:00:08.191579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191540    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kube-rbac-proxy"
Jul 30 18:00:08.191579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191554    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container"
Jul 30 18:00:08.191579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191560    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container"
Jul 30 18:00:08.191579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191567    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="storage-initializer"
Jul 30 18:00:08.191579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191573    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="storage-initializer"
Jul 30 18:00:08.191729 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191612    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kserve-container"
Jul 30 18:00:08.191729 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.191621    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" containerName="kube-rbac-proxy"
Jul 30 18:00:08.194787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.194771    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.197107 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.197083    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 30 18:00:08.197107 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.197092    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 30 18:00:08.204106 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.204081    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"]
Jul 30 18:00:08.258300 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.258270    2569 generic.go:356] "Generic (PLEG): container finished" podID="daf96f80-b682-412f-b46d-51b2ff803a13" containerID="02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313" exitCode=0
Jul 30 18:00:08.258455 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.258338    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"
Jul 30 18:00:08.258455 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.258345    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerDied","Data":"02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313"}
Jul 30 18:00:08.258455 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.258403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm" event={"ID":"daf96f80-b682-412f-b46d-51b2ff803a13","Type":"ContainerDied","Data":"d9efd32487adc82765e366abb2bd6eb27e4d51b1cbe43c19d3d9e5b1cc5b75bb"}
Jul 30 18:00:08.258455 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.258434    2569 scope.go:117] "RemoveContainer" containerID="b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e"
Jul 30 18:00:08.266207 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.266088    2569 scope.go:117] "RemoveContainer" containerID="02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313"
Jul 30 18:00:08.273117 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.273101    2569 scope.go:117] "RemoveContainer" containerID="a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab"
Jul 30 18:00:08.279718 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.279668    2569 scope.go:117] "RemoveContainer" containerID="b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e"
Jul 30 18:00:08.280327 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:00:08.280300    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e\": container with ID starting with b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e not found: ID does not exist" containerID="b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e"
Jul 30 18:00:08.280437 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.280334    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e"} err="failed to get container status \"b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e\": rpc error: code = NotFound desc = could not find container \"b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e\": container with ID starting with b2750ced5fe04b908e014ce399facb932b66c33135d5a0dea22104e9f47d0f1e not found: ID does not exist"
Jul 30 18:00:08.280437 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.280360    2569 scope.go:117] "RemoveContainer" containerID="02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313"
Jul 30 18:00:08.280647 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:00:08.280628    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313\": container with ID starting with 02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313 not found: ID does not exist" containerID="02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313"
Jul 30 18:00:08.280691 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.280654    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313"} err="failed to get container status \"02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313\": rpc error: code = NotFound desc = could not find container \"02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313\": container with ID starting with 02cf9667a54f559a43e66cbbf7e51d5e06b9cbe65c6af901d62ae3a263405313 not found: ID does not exist"
Jul 30 18:00:08.280691 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.280669    2569 scope.go:117] "RemoveContainer" containerID="a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab"
Jul 30 18:00:08.280939 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:00:08.280918    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab\": container with ID starting with a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab not found: ID does not exist" containerID="a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab"
Jul 30 18:00:08.280984 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.280947    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab"} err="failed to get container status \"a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab\": rpc error: code = NotFound desc = could not find container \"a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab\": container with ID starting with a310007efbcfb051871b7860a7b5f32174873657cc1f19217572c7b4b84854ab not found: ID does not exist"
Jul 30 18:00:08.281741 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.281721    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"]
Jul 30 18:00:08.287555 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.287537    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-8g2xm"]
Jul 30 18:00:08.305037 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.305019    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.305127 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.305051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784c976e-e7c5-4560-ab13-0c2b23306432-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.305127 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.305073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96qv\" (UniqueName: \"kubernetes.io/projected/784c976e-e7c5-4560-ab13-0c2b23306432-kube-api-access-r96qv\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.305127 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.305103    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.305307 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.305165    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784c976e-e7c5-4560-ab13-0c2b23306432-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406121 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96qv\" (UniqueName: \"kubernetes.io/projected/784c976e-e7c5-4560-ab13-0c2b23306432-kube-api-access-r96qv\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406121 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406342 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784c976e-e7c5-4560-ab13-0c2b23306432-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406342 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406199    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406342 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406223    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784c976e-e7c5-4560-ab13-0c2b23306432-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406770 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406729    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784c976e-e7c5-4560-ab13-0c2b23306432-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406834 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406811    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.406967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.406951    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.408746 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.408721    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784c976e-e7c5-4560-ab13-0c2b23306432-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.415577 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.415555    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96qv\" (UniqueName: \"kubernetes.io/projected/784c976e-e7c5-4560-ab13-0c2b23306432-kube-api-access-r96qv\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.505354 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.505301    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:08.633436 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.633198    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"]
Jul 30 18:00:08.635963 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:00:08.635933    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784c976e_e7c5_4560_ab13_0c2b23306432.slice/crio-d55a73e6ca6bfd304f9c6628970fc734521aac8660990115556f261a4db0b49e WatchSource:0}: Error finding container d55a73e6ca6bfd304f9c6628970fc734521aac8660990115556f261a4db0b49e: Status 404 returned error can't find the container with id d55a73e6ca6bfd304f9c6628970fc734521aac8660990115556f261a4db0b49e
Jul 30 18:00:08.651009 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:08.650984    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf96f80-b682-412f-b46d-51b2ff803a13" path="/var/lib/kubelet/pods/daf96f80-b682-412f-b46d-51b2ff803a13/volumes"
Jul 30 18:00:09.270070 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:09.269891    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerStarted","Data":"e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314"}
Jul 30 18:00:09.270510 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:09.270123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerStarted","Data":"d55a73e6ca6bfd304f9c6628970fc734521aac8660990115556f261a4db0b49e"}
Jul 30 18:00:13.286001 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:13.285967    2569 generic.go:356] "Generic (PLEG): container finished" podID="784c976e-e7c5-4560-ab13-0c2b23306432" containerID="e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314" exitCode=0
Jul 30 18:00:13.286402 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:13.286019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerDied","Data":"e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314"}
Jul 30 18:00:14.291152 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:14.291118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerStarted","Data":"306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093"}
Jul 30 18:00:14.291582 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:14.291182    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerStarted","Data":"3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5"}
Jul 30 18:00:14.291582 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:14.291415    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:14.311337 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:14.311288    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podStartSLOduration=2.510444723 podStartE2EDuration="6.311274517s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800829794s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:00:14.310147423 +0000 UTC m=+3582.342957753" watchObservedRunningTime="2026-07-30 18:00:14.311274517 +0000 UTC m=+3582.344084847"
Jul 30 18:00:15.295092 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:15.295056    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:15.296427 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:15.296394    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:00:16.298269 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:16.298207    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:00:21.304245 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:21.304211    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:00:21.305337 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:21.305303    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:00:31.305210 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:31.305170    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:00:41.305409 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:41.305365    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:00:51.305405 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:00:51.305359    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:01:01.306061 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:01.306016    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:01:11.305459 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:11.305418    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:01:21.305419 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:21.305381    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:01:31.305984 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:31.305951    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:01:38.306217 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:38.306183    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"]
Jul 30 18:01:38.306621 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:38.306498    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" containerID="cri-o://3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5" gracePeriod=30
Jul 30 18:01:38.306671 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:38.306586    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kube-rbac-proxy" containerID="cri-o://306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093" gracePeriod=30
Jul 30 18:01:38.621053 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:38.620957    2569 generic.go:356] "Generic (PLEG): container finished" podID="784c976e-e7c5-4560-ab13-0c2b23306432" containerID="306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093" exitCode=2
Jul 30 18:01:38.621053 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:38.621007    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerDied","Data":"306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093"}
Jul 30 18:01:41.299309 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:41.299244    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 30 18:01:41.305698 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:41.305664    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 30 18:01:43.246664 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.246636    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:01:43.339038 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339008    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784c976e-e7c5-4560-ab13-0c2b23306432-kserve-provision-location\") pod \"784c976e-e7c5-4560-ab13-0c2b23306432\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") "
Jul 30 18:01:43.339228 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-cabundle-cert\") pod \"784c976e-e7c5-4560-ab13-0c2b23306432\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") "
Jul 30 18:01:43.339228 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339068    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-kube-rbac-proxy-sar-config\") pod \"784c976e-e7c5-4560-ab13-0c2b23306432\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") "
Jul 30 18:01:43.339228 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339120    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784c976e-e7c5-4560-ab13-0c2b23306432-proxy-tls\") pod \"784c976e-e7c5-4560-ab13-0c2b23306432\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") "
Jul 30 18:01:43.339228 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339181    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r96qv\" (UniqueName: \"kubernetes.io/projected/784c976e-e7c5-4560-ab13-0c2b23306432-kube-api-access-r96qv\") pod \"784c976e-e7c5-4560-ab13-0c2b23306432\" (UID: \"784c976e-e7c5-4560-ab13-0c2b23306432\") "
Jul 30 18:01:43.339474 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339437    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784c976e-e7c5-4560-ab13-0c2b23306432-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "784c976e-e7c5-4560-ab13-0c2b23306432" (UID: "784c976e-e7c5-4560-ab13-0c2b23306432"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:01:43.339474 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339452    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "784c976e-e7c5-4560-ab13-0c2b23306432" (UID: "784c976e-e7c5-4560-ab13-0c2b23306432"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:01:43.339558 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.339494    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "784c976e-e7c5-4560-ab13-0c2b23306432" (UID: "784c976e-e7c5-4560-ab13-0c2b23306432"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:01:43.341416 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.341385    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784c976e-e7c5-4560-ab13-0c2b23306432-kube-api-access-r96qv" (OuterVolumeSpecName: "kube-api-access-r96qv") pod "784c976e-e7c5-4560-ab13-0c2b23306432" (UID: "784c976e-e7c5-4560-ab13-0c2b23306432"). InnerVolumeSpecName "kube-api-access-r96qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:01:43.341534 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.341494    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784c976e-e7c5-4560-ab13-0c2b23306432-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "784c976e-e7c5-4560-ab13-0c2b23306432" (UID: "784c976e-e7c5-4560-ab13-0c2b23306432"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:01:43.440514 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.440480    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/784c976e-e7c5-4560-ab13-0c2b23306432-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:01:43.440514 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.440511    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96qv\" (UniqueName: \"kubernetes.io/projected/784c976e-e7c5-4560-ab13-0c2b23306432-kube-api-access-r96qv\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:01:43.440708 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.440521    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/784c976e-e7c5-4560-ab13-0c2b23306432-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:01:43.440708 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.440530    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:01:43.440708 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.440539    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/784c976e-e7c5-4560-ab13-0c2b23306432-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:01:43.490042 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490010    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"]
Jul 30 18:01:43.490299 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490285    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kube-rbac-proxy"
Jul 30 18:01:43.490354 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490301    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kube-rbac-proxy"
Jul 30 18:01:43.490354 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490314    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container"
Jul 30 18:01:43.490354 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490320    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container"
Jul 30 18:01:43.490354 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490334    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="storage-initializer"
Jul 30 18:01:43.490354 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490340    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="storage-initializer"
Jul 30 18:01:43.490505 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490393    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kube-rbac-proxy"
Jul 30 18:01:43.490505 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.490402    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" containerName="kserve-container"
Jul 30 18:01:43.493640 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.493623    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.495859 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.495838    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 30 18:01:43.495964 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.495919    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 30 18:01:43.504566 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.504544    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"]
Jul 30 18:01:43.641959 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.641924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/193be677-a51e-4ed8-b2c5-a292891869c2-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.641959 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.641962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.642206 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/193be677-a51e-4ed8-b2c5-a292891869c2-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.642206 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642047    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.642206 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642108    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf8b5\" (UniqueName: \"kubernetes.io/projected/193be677-a51e-4ed8-b2c5-a292891869c2-kube-api-access-sf8b5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.642421 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642402    2569 generic.go:356] "Generic (PLEG): container finished" podID="784c976e-e7c5-4560-ab13-0c2b23306432" containerID="3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5" exitCode=0
Jul 30 18:01:43.642486 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642471    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"
Jul 30 18:01:43.642486 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerDied","Data":"3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5"}
Jul 30 18:01:43.642590 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw" event={"ID":"784c976e-e7c5-4560-ab13-0c2b23306432","Type":"ContainerDied","Data":"d55a73e6ca6bfd304f9c6628970fc734521aac8660990115556f261a4db0b49e"}
Jul 30 18:01:43.642590 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.642584    2569 scope.go:117] "RemoveContainer" containerID="306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093"
Jul 30 18:01:43.651469 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.651452    2569 scope.go:117] "RemoveContainer" containerID="3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5"
Jul 30 18:01:43.658614 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.658595    2569 scope.go:117] "RemoveContainer" containerID="e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314"
Jul 30 18:01:43.665302 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.665282    2569 scope.go:117] "RemoveContainer" containerID="306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093"
Jul 30 18:01:43.665553 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:01:43.665535    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093\": container with ID starting with 306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093 not found: ID does not exist" containerID="306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093"
Jul 30 18:01:43.665636 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.665569    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093"} err="failed to get container status \"306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093\": rpc error: code = NotFound desc = could not find container \"306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093\": container with ID starting with 306a243c99e6064b64a55980899542102c1779a6df73cc367dec2b10da1ef093 not found: ID does not exist"
Jul 30 18:01:43.665636 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.665590    2569 scope.go:117] "RemoveContainer" containerID="3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5"
Jul 30 18:01:43.665888 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:01:43.665844    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5\": container with ID starting with 3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5 not found: ID does not exist" containerID="3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5"
Jul 30 18:01:43.665888 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.665875    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5"} err="failed to get container status \"3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5\": rpc error: code = NotFound desc = could not find container \"3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5\": container with ID starting with 3617661e5a6b09edde1c5aea77017684adfe56ad1f872155e5b27ccc138db8a5 not found: ID does not exist"
Jul 30 18:01:43.666441 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.665896    2569 scope.go:117] "RemoveContainer" containerID="e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314"
Jul 30 18:01:43.666497 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:01:43.666431    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314\": container with ID starting with e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314 not found: ID does not exist" containerID="e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314"
Jul 30 18:01:43.666497 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.666462    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314"} err="failed to get container status \"e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314\": rpc error: code = NotFound desc = could not find container \"e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314\": container with ID starting with e89d5298c929e98a925f4a8aa98ca4ee594370732733bcefa23629e958e95314 not found: ID does not exist"
Jul 30 18:01:43.667739 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.667720    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"]
Jul 30 18:01:43.672323 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.672305    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-pjlxw"]
Jul 30 18:01:43.742684 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.742631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf8b5\" (UniqueName: \"kubernetes.io/projected/193be677-a51e-4ed8-b2c5-a292891869c2-kube-api-access-sf8b5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.742770 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.742722    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/193be677-a51e-4ed8-b2c5-a292891869c2-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.742770 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.742746    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.742869 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.742822    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/193be677-a51e-4ed8-b2c5-a292891869c2-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.742869 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.742849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.743140 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.743119    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/193be677-a51e-4ed8-b2c5-a292891869c2-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.743448 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.743429    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.743448 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.743443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.745304 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.745283    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/193be677-a51e-4ed8-b2c5-a292891869c2-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.751690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.751665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf8b5\" (UniqueName: \"kubernetes.io/projected/193be677-a51e-4ed8-b2c5-a292891869c2-kube-api-access-sf8b5\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.804097 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.804071    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:43.934875 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:43.934842    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"]
Jul 30 18:01:43.938366 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:01:43.938333    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193be677_a51e_4ed8_b2c5_a292891869c2.slice/crio-3a89564f922ed6eeb7fc25301189f61cfc067c076c9314bfe33d331449870a99 WatchSource:0}: Error finding container 3a89564f922ed6eeb7fc25301189f61cfc067c076c9314bfe33d331449870a99: Status 404 returned error can't find the container with id 3a89564f922ed6eeb7fc25301189f61cfc067c076c9314bfe33d331449870a99
Jul 30 18:01:44.650852 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:44.650817    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784c976e-e7c5-4560-ab13-0c2b23306432" path="/var/lib/kubelet/pods/784c976e-e7c5-4560-ab13-0c2b23306432/volumes"
Jul 30 18:01:44.651216 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:44.651130    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerStarted","Data":"3dba387c6f2ed1d299cdc81abbfbd65e57490980ae3c678950a1c1cd4ce87eb0"}
Jul 30 18:01:44.651216 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:44.651175    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerStarted","Data":"3a89564f922ed6eeb7fc25301189f61cfc067c076c9314bfe33d331449870a99"}
Jul 30 18:01:47.660223 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:47.660184    2569 generic.go:356] "Generic (PLEG): container finished" podID="193be677-a51e-4ed8-b2c5-a292891869c2" containerID="3dba387c6f2ed1d299cdc81abbfbd65e57490980ae3c678950a1c1cd4ce87eb0" exitCode=0
Jul 30 18:01:47.660562 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:47.660282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerDied","Data":"3dba387c6f2ed1d299cdc81abbfbd65e57490980ae3c678950a1c1cd4ce87eb0"}
Jul 30 18:01:48.665498 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:48.665466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerStarted","Data":"31d2bef2e84501e7a675ed9ec1865d7971507059c55cbaaf9be9de9293e3d796"}
Jul 30 18:01:48.665872 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:48.665525    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerStarted","Data":"280563fa0e4c93f3719798b7393a080bb1bbad6178320ed4f39c0eefe2f9fdf6"}
Jul 30 18:01:48.665872 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:48.665682    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:48.685379 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:48.685341    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podStartSLOduration=2.091530803 podStartE2EDuration="5.685328254s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.593797451s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:01:48.684149157 +0000 UTC m=+3676.716959758" watchObservedRunningTime="2026-07-30 18:01:48.685328254 +0000 UTC m=+3676.718138569"
Jul 30 18:01:49.670056 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:49.670026    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:01:55.679646 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:01:55.679555    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:02:25.680861 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:02:25.680818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 30 18:02:35.680946 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:02:35.680901    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 30 18:02:45.680496 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:02:45.680457    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 30 18:02:55.680882 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:02:55.680843    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.45:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 30 18:03:00.722568 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:00.722530    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:03:04.373684 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:04.373642    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"]
Jul 30 18:03:04.374102 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:04.373868    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container" containerID="cri-o://280563fa0e4c93f3719798b7393a080bb1bbad6178320ed4f39c0eefe2f9fdf6" gracePeriod=30
Jul 30 18:03:04.374102 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:04.373905    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kube-rbac-proxy" containerID="cri-o://31d2bef2e84501e7a675ed9ec1865d7971507059c55cbaaf9be9de9293e3d796" gracePeriod=30
Jul 30 18:03:04.952898 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:04.952865    2569 generic.go:356] "Generic (PLEG): container finished" podID="193be677-a51e-4ed8-b2c5-a292891869c2" containerID="31d2bef2e84501e7a675ed9ec1865d7971507059c55cbaaf9be9de9293e3d796" exitCode=2
Jul 30 18:03:04.953070 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:04.952932    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerDied","Data":"31d2bef2e84501e7a675ed9ec1865d7971507059c55cbaaf9be9de9293e3d796"}
Jul 30 18:03:05.674681 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:05.674640    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 30 18:03:08.969250 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:08.969209    2569 generic.go:356] "Generic (PLEG): container finished" podID="193be677-a51e-4ed8-b2c5-a292891869c2" containerID="280563fa0e4c93f3719798b7393a080bb1bbad6178320ed4f39c0eefe2f9fdf6" exitCode=0
Jul 30 18:03:08.969589 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:08.969289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerDied","Data":"280563fa0e4c93f3719798b7393a080bb1bbad6178320ed4f39c0eefe2f9fdf6"}
Jul 30 18:03:09.014484 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.014461    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:03:09.115317 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115214    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-kube-rbac-proxy-sar-config\") pod \"193be677-a51e-4ed8-b2c5-a292891869c2\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") "
Jul 30 18:03:09.115317 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115280    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/193be677-a51e-4ed8-b2c5-a292891869c2-kserve-provision-location\") pod \"193be677-a51e-4ed8-b2c5-a292891869c2\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") "
Jul 30 18:03:09.115317 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115315    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf8b5\" (UniqueName: \"kubernetes.io/projected/193be677-a51e-4ed8-b2c5-a292891869c2-kube-api-access-sf8b5\") pod \"193be677-a51e-4ed8-b2c5-a292891869c2\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") "
Jul 30 18:03:09.115548 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115359    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-cabundle-cert\") pod \"193be677-a51e-4ed8-b2c5-a292891869c2\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") "
Jul 30 18:03:09.115548 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/193be677-a51e-4ed8-b2c5-a292891869c2-proxy-tls\") pod \"193be677-a51e-4ed8-b2c5-a292891869c2\" (UID: \"193be677-a51e-4ed8-b2c5-a292891869c2\") "
Jul 30 18:03:09.115670 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115606    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193be677-a51e-4ed8-b2c5-a292891869c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "193be677-a51e-4ed8-b2c5-a292891869c2" (UID: "193be677-a51e-4ed8-b2c5-a292891869c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:03:09.115725 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115640    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "193be677-a51e-4ed8-b2c5-a292891869c2" (UID: "193be677-a51e-4ed8-b2c5-a292891869c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:03:09.115782 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.115767    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "193be677-a51e-4ed8-b2c5-a292891869c2" (UID: "193be677-a51e-4ed8-b2c5-a292891869c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:03:09.117505 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.117476    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193be677-a51e-4ed8-b2c5-a292891869c2-kube-api-access-sf8b5" (OuterVolumeSpecName: "kube-api-access-sf8b5") pod "193be677-a51e-4ed8-b2c5-a292891869c2" (UID: "193be677-a51e-4ed8-b2c5-a292891869c2"). InnerVolumeSpecName "kube-api-access-sf8b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:03:09.117656 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.117568    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193be677-a51e-4ed8-b2c5-a292891869c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "193be677-a51e-4ed8-b2c5-a292891869c2" (UID: "193be677-a51e-4ed8-b2c5-a292891869c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:03:09.216122 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.216069    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:03:09.216122 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.216114    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/193be677-a51e-4ed8-b2c5-a292891869c2-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:03:09.216122 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.216124    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf8b5\" (UniqueName: \"kubernetes.io/projected/193be677-a51e-4ed8-b2c5-a292891869c2-kube-api-access-sf8b5\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:03:09.216122 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.216134    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/193be677-a51e-4ed8-b2c5-a292891869c2-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:03:09.216122 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.216144    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/193be677-a51e-4ed8-b2c5-a292891869c2-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:03:09.540544 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540508    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"]
Jul 30 18:03:09.540773 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540761    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kube-rbac-proxy"
Jul 30 18:03:09.540832 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540773    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kube-rbac-proxy"
Jul 30 18:03:09.540832 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540787    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="storage-initializer"
Jul 30 18:03:09.540832 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540793    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="storage-initializer"
Jul 30 18:03:09.540832 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540798    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container"
Jul 30 18:03:09.540832 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540804    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container"
Jul 30 18:03:09.541004 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540850    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kserve-container"
Jul 30 18:03:09.541004 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.540858    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" containerName="kube-rbac-proxy"
Jul 30 18:03:09.543827 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.543811    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.546336 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.546310    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 30 18:03:09.546437 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.546315    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 30 18:03:09.555723 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.555697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"]
Jul 30 18:03:09.619449 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.619415    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.619631 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.619462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387fb8a-291d-4cb7-8fff-6184036ae973-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.619631 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.619484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfzlk\" (UniqueName: \"kubernetes.io/projected/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-api-access-xfzlk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.619631 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.619605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.619777 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.619667    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387fb8a-291d-4cb7-8fff-6184036ae973-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720058 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720069    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387fb8a-291d-4cb7-8fff-6184036ae973-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfzlk\" (UniqueName: \"kubernetes.io/projected/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-api-access-xfzlk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720276    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720476 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387fb8a-291d-4cb7-8fff-6184036ae973-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720671 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387fb8a-291d-4cb7-8fff-6184036ae973-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720776 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720758    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.720846 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.720827    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.722587 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.722569    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387fb8a-291d-4cb7-8fff-6184036ae973-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.730009 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.729990    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfzlk\" (UniqueName: \"kubernetes.io/projected/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-api-access-xfzlk\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.853891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.853803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:09.974792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.974761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l" event={"ID":"193be677-a51e-4ed8-b2c5-a292891869c2","Type":"ContainerDied","Data":"3a89564f922ed6eeb7fc25301189f61cfc067c076c9314bfe33d331449870a99"}
Jul 30 18:03:09.975146 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.974819    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"
Jul 30 18:03:09.975146 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.974831    2569 scope.go:117] "RemoveContainer" containerID="31d2bef2e84501e7a675ed9ec1865d7971507059c55cbaaf9be9de9293e3d796"
Jul 30 18:03:09.982736 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.982597    2569 scope.go:117] "RemoveContainer" containerID="280563fa0e4c93f3719798b7393a080bb1bbad6178320ed4f39c0eefe2f9fdf6"
Jul 30 18:03:09.987760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.987739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"]
Jul 30 18:03:09.993338 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:09.993311    2569 scope.go:117] "RemoveContainer" containerID="3dba387c6f2ed1d299cdc81abbfbd65e57490980ae3c678950a1c1cd4ce87eb0"
Jul 30 18:03:09.994509 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:03:09.994488    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4387fb8a_291d_4cb7_8fff_6184036ae973.slice/crio-983e539d0ba785b64538070d37a419b23fe8899fa166eda27fc20b76c8e7bb50 WatchSource:0}: Error finding container 983e539d0ba785b64538070d37a419b23fe8899fa166eda27fc20b76c8e7bb50: Status 404 returned error can't find the container with id 983e539d0ba785b64538070d37a419b23fe8899fa166eda27fc20b76c8e7bb50
Jul 30 18:03:10.002623 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:10.002601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"]
Jul 30 18:03:10.008460 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:10.008440    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-dwr6l"]
Jul 30 18:03:10.651739 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:10.651698    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="193be677-a51e-4ed8-b2c5-a292891869c2" path="/var/lib/kubelet/pods/193be677-a51e-4ed8-b2c5-a292891869c2/volumes"
Jul 30 18:03:10.980635 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:10.980594    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerStarted","Data":"83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0"}
Jul 30 18:03:10.981075 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:10.980665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerStarted","Data":"983e539d0ba785b64538070d37a419b23fe8899fa166eda27fc20b76c8e7bb50"}
Jul 30 18:03:13.994473 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:13.994439    2569 generic.go:356] "Generic (PLEG): container finished" podID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerID="83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0" exitCode=0
Jul 30 18:03:13.994941 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:13.994520    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerDied","Data":"83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0"}
Jul 30 18:03:15.000298 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:15.000247    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerStarted","Data":"0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9"}
Jul 30 18:03:15.000707 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:15.000324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerStarted","Data":"b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea"}
Jul 30 18:03:15.000707 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:15.000602    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:15.000707 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:15.000666    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:15.021965 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:15.021918    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podStartSLOduration=2.280297323 podStartE2EDuration="6.021902603s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.74160528s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:03:15.020821297 +0000 UTC m=+3763.053631635" watchObservedRunningTime="2026-07-30 18:03:15.021902603 +0000 UTC m=+3763.054712939"
Jul 30 18:03:21.009996 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:21.009968    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:03:51.011369 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:03:51.011329    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 30 18:04:01.010696 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:01.010652    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 30 18:04:11.011274 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:11.011223    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 30 18:04:21.011067 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:21.011022    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 30 18:04:31.028119 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:31.028086    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:04:39.718335 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:39.718299    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"]
Jul 30 18:04:39.718802 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:39.718602    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container" containerID="cri-o://b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea" gracePeriod=30
Jul 30 18:04:39.718802 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:39.718635    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kube-rbac-proxy" containerID="cri-o://0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9" gracePeriod=30
Jul 30 18:04:40.327937 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:40.327905    2569 generic.go:356] "Generic (PLEG): container finished" podID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerID="0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9" exitCode=2
Jul 30 18:04:40.328090 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:40.327959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerDied","Data":"0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9"}
Jul 30 18:04:41.005006 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:41.004960    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 30 18:04:41.010760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:41.010727    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 30 18:04:44.263749 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.263726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:04:44.335602 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335574    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-cabundle-cert\") pod \"4387fb8a-291d-4cb7-8fff-6184036ae973\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") "
Jul 30 18:04:44.335602 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335609    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-rbac-proxy-sar-config\") pod \"4387fb8a-291d-4cb7-8fff-6184036ae973\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") "
Jul 30 18:04:44.335867 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335640    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfzlk\" (UniqueName: \"kubernetes.io/projected/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-api-access-xfzlk\") pod \"4387fb8a-291d-4cb7-8fff-6184036ae973\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") "
Jul 30 18:04:44.335867 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387fb8a-291d-4cb7-8fff-6184036ae973-kserve-provision-location\") pod \"4387fb8a-291d-4cb7-8fff-6184036ae973\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") "
Jul 30 18:04:44.335867 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335840    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387fb8a-291d-4cb7-8fff-6184036ae973-proxy-tls\") pod \"4387fb8a-291d-4cb7-8fff-6184036ae973\" (UID: \"4387fb8a-291d-4cb7-8fff-6184036ae973\") "
Jul 30 18:04:44.336032 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335975    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387fb8a-291d-4cb7-8fff-6184036ae973-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4387fb8a-291d-4cb7-8fff-6184036ae973" (UID: "4387fb8a-291d-4cb7-8fff-6184036ae973"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:04:44.336032 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.335982    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4387fb8a-291d-4cb7-8fff-6184036ae973" (UID: "4387fb8a-291d-4cb7-8fff-6184036ae973"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:04:44.336111 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.336033    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4387fb8a-291d-4cb7-8fff-6184036ae973" (UID: "4387fb8a-291d-4cb7-8fff-6184036ae973"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:04:44.336162 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.336142    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387fb8a-291d-4cb7-8fff-6184036ae973-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:04:44.336287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.336170    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:04:44.336287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.336183    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:04:44.337753 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.337730    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-api-access-xfzlk" (OuterVolumeSpecName: "kube-api-access-xfzlk") pod "4387fb8a-291d-4cb7-8fff-6184036ae973" (UID: "4387fb8a-291d-4cb7-8fff-6184036ae973"). InnerVolumeSpecName "kube-api-access-xfzlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:04:44.337911 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.337883    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4387fb8a-291d-4cb7-8fff-6184036ae973-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4387fb8a-291d-4cb7-8fff-6184036ae973" (UID: "4387fb8a-291d-4cb7-8fff-6184036ae973"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:04:44.345958 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.345932    2569 generic.go:356] "Generic (PLEG): container finished" podID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerID="b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea" exitCode=0
Jul 30 18:04:44.346084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.346002    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"
Jul 30 18:04:44.346084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.346008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerDied","Data":"b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea"}
Jul 30 18:04:44.346084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.346068    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg" event={"ID":"4387fb8a-291d-4cb7-8fff-6184036ae973","Type":"ContainerDied","Data":"983e539d0ba785b64538070d37a419b23fe8899fa166eda27fc20b76c8e7bb50"}
Jul 30 18:04:44.346273 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.346096    2569 scope.go:117] "RemoveContainer" containerID="0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9"
Jul 30 18:04:44.353895 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.353877    2569 scope.go:117] "RemoveContainer" containerID="b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea"
Jul 30 18:04:44.361159 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.361134    2569 scope.go:117] "RemoveContainer" containerID="83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0"
Jul 30 18:04:44.368080 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.368060    2569 scope.go:117] "RemoveContainer" containerID="0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9"
Jul 30 18:04:44.368343 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:04:44.368321    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9\": container with ID starting with 0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9 not found: ID does not exist" containerID="0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9"
Jul 30 18:04:44.368397 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.368351    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9"} err="failed to get container status \"0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9\": rpc error: code = NotFound desc = could not find container \"0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9\": container with ID starting with 0f591faaf6d46707f243f4af56bcb6a6bf90dfa53812d076eb7c50a3d810f1b9 not found: ID does not exist"
Jul 30 18:04:44.368397 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.368369    2569 scope.go:117] "RemoveContainer" containerID="b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea"
Jul 30 18:04:44.368581 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:04:44.368561    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea\": container with ID starting with b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea not found: ID does not exist" containerID="b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea"
Jul 30 18:04:44.368651 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.368591    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea"} err="failed to get container status \"b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea\": rpc error: code = NotFound desc = could not find container \"b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea\": container with ID starting with b54aa2d117d45eacbd778205f3367992dcae2e1dcdf1c2cec0271c68724613ea not found: ID does not exist"
Jul 30 18:04:44.368651 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.368614    2569 scope.go:117] "RemoveContainer" containerID="83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0"
Jul 30 18:04:44.368873 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:04:44.368856    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0\": container with ID starting with 83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0 not found: ID does not exist" containerID="83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0"
Jul 30 18:04:44.368912 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.368878    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0"} err="failed to get container status \"83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0\": rpc error: code = NotFound desc = could not find container \"83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0\": container with ID starting with 83c9bda39aa52226a68c220e99e6b580d9f2100cea8f93fe06490264e34587d0 not found: ID does not exist"
Jul 30 18:04:44.371890 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.371838    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"]
Jul 30 18:04:44.376541 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.376522    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-nl2jg"]
Jul 30 18:04:44.436601 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.436565    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfzlk\" (UniqueName: \"kubernetes.io/projected/4387fb8a-291d-4cb7-8fff-6184036ae973-kube-api-access-xfzlk\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:04:44.436601 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.436593    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387fb8a-291d-4cb7-8fff-6184036ae973-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:04:44.651708 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.651630    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" path="/var/lib/kubelet/pods/4387fb8a-291d-4cb7-8fff-6184036ae973/volumes"
Jul 30 18:04:44.860346 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860314    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"]
Jul 30 18:04:44.860590 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860579    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="storage-initializer"
Jul 30 18:04:44.860635 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860591    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="storage-initializer"
Jul 30 18:04:44.860635 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860611    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container"
Jul 30 18:04:44.860635 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860616    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container"
Jul 30 18:04:44.860635 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860624    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kube-rbac-proxy"
Jul 30 18:04:44.860635 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860630    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kube-rbac-proxy"
Jul 30 18:04:44.860784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860672    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kube-rbac-proxy"
Jul 30 18:04:44.860784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.860680    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4387fb8a-291d-4cb7-8fff-6184036ae973" containerName="kserve-container"
Jul 30 18:04:44.865389 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.865371    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:44.868438 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868409    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:04:44.868572 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868452    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 30 18:04:44.868572 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868473    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:04:44.868690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868590    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:04:44.868690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868599    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:04:44.868690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868632    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 30 18:04:44.868690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.868593    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:04:44.876176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.876151    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"]
Jul 30 18:04:44.940375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.940337    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:44.940552 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.940391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97c2bc4-3446-4535-8566-9f3847b59822-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:44.940552 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.940476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97c2bc4-3446-4535-8566-9f3847b59822-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:44.940552 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.940510    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwshh\" (UniqueName: \"kubernetes.io/projected/b97c2bc4-3446-4535-8566-9f3847b59822-kube-api-access-rwshh\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:44.940552 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:44.940534    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.040876 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.040842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97c2bc4-3446-4535-8566-9f3847b59822-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.040876 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.040878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwshh\" (UniqueName: \"kubernetes.io/projected/b97c2bc4-3446-4535-8566-9f3847b59822-kube-api-access-rwshh\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.041084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.040904    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.041084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.040972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.041084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.041009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97c2bc4-3446-4535-8566-9f3847b59822-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.041454 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.041428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97c2bc4-3446-4535-8566-9f3847b59822-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.041776 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.041753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.041776 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.041766    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.043291 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.043275    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97c2bc4-3446-4535-8566-9f3847b59822-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.050860 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.050838    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwshh\" (UniqueName: \"kubernetes.io/projected/b97c2bc4-3446-4535-8566-9f3847b59822-kube-api-access-rwshh\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.175789 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.175755    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:45.305675 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.305644    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"]
Jul 30 18:04:45.308852 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:04:45.308816    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97c2bc4_3446_4535_8566_9f3847b59822.slice/crio-6ea191802e5c10c466c956245ac7951c169413963456a549e93e72018725413b WatchSource:0}: Error finding container 6ea191802e5c10c466c956245ac7951c169413963456a549e93e72018725413b: Status 404 returned error can't find the container with id 6ea191802e5c10c466c956245ac7951c169413963456a549e93e72018725413b
Jul 30 18:04:45.355945 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:45.355916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerStarted","Data":"6ea191802e5c10c466c956245ac7951c169413963456a549e93e72018725413b"}
Jul 30 18:04:46.361146 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:46.361107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerStarted","Data":"e29ef8abf72893771902935f487d01aedee97a5678a3c6de077084e7c7c484e8"}
Jul 30 18:04:49.374087 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:49.374049    2569 generic.go:356] "Generic (PLEG): container finished" podID="b97c2bc4-3446-4535-8566-9f3847b59822" containerID="e29ef8abf72893771902935f487d01aedee97a5678a3c6de077084e7c7c484e8" exitCode=0
Jul 30 18:04:49.374504 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:49.374107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerDied","Data":"e29ef8abf72893771902935f487d01aedee97a5678a3c6de077084e7c7c484e8"}
Jul 30 18:04:50.380437 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:50.380403    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerStarted","Data":"2dde32f3ca2656d22b435fa25793d6a8927f22eaf5140bca67777797d5e910a9"}
Jul 30 18:04:50.380826 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:50.380463    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerStarted","Data":"cb2a2e3b91e5e0fada8d9f311f7a30a2de6b96bcf6c3e73dd728f7e3848ba44e"}
Jul 30 18:04:50.380826 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:50.380737    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:50.380826 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:50.380803    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:04:50.416551 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:50.416505    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podStartSLOduration=2.636007994 podStartE2EDuration="6.41648898s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780480986s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:04:50.416028131 +0000 UTC m=+3858.448838468" watchObservedRunningTime="2026-07-30 18:04:50.41648898 +0000 UTC m=+3858.449299317"
Jul 30 18:04:56.390770 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:04:56.390695    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:05:26.391856 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:05:26.391817    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 30 18:05:36.392154 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:05:36.392110    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 30 18:05:46.392397 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:05:46.392346    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 30 18:05:56.391669 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:05:56.391627    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.47:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 30 18:06:04.720350 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:04.720317    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:06:14.949546 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:14.949497    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"]
Jul 30 18:06:14.950020 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:14.949751    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container" containerID="cri-o://cb2a2e3b91e5e0fada8d9f311f7a30a2de6b96bcf6c3e73dd728f7e3848ba44e" gracePeriod=30
Jul 30 18:06:14.950020 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:14.949782    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kube-rbac-proxy" containerID="cri-o://2dde32f3ca2656d22b435fa25793d6a8927f22eaf5140bca67777797d5e910a9" gracePeriod=30
Jul 30 18:06:15.702392 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:15.702356    2569 generic.go:356] "Generic (PLEG): container finished" podID="b97c2bc4-3446-4535-8566-9f3847b59822" containerID="2dde32f3ca2656d22b435fa25793d6a8927f22eaf5140bca67777797d5e910a9" exitCode=2
Jul 30 18:06:15.702564 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:15.702427    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerDied","Data":"2dde32f3ca2656d22b435fa25793d6a8927f22eaf5140bca67777797d5e910a9"}
Jul 30 18:06:16.385621 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:16.385579    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 30 18:06:19.718316 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:19.718284    2569 generic.go:356] "Generic (PLEG): container finished" podID="b97c2bc4-3446-4535-8566-9f3847b59822" containerID="cb2a2e3b91e5e0fada8d9f311f7a30a2de6b96bcf6c3e73dd728f7e3848ba44e" exitCode=0
Jul 30 18:06:19.718690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:19.718367    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerDied","Data":"cb2a2e3b91e5e0fada8d9f311f7a30a2de6b96bcf6c3e73dd728f7e3848ba44e"}
Jul 30 18:06:19.992100 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:19.992076    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:06:20.065027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.064992    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97c2bc4-3446-4535-8566-9f3847b59822-proxy-tls\") pod \"b97c2bc4-3446-4535-8566-9f3847b59822\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") "
Jul 30 18:06:20.065200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065036    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97c2bc4-3446-4535-8566-9f3847b59822-kserve-provision-location\") pod \"b97c2bc4-3446-4535-8566-9f3847b59822\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") "
Jul 30 18:06:20.065200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065066    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-kube-rbac-proxy-sar-config\") pod \"b97c2bc4-3446-4535-8566-9f3847b59822\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") "
Jul 30 18:06:20.065200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065089    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-cabundle-cert\") pod \"b97c2bc4-3446-4535-8566-9f3847b59822\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") "
Jul 30 18:06:20.065200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065125    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwshh\" (UniqueName: \"kubernetes.io/projected/b97c2bc4-3446-4535-8566-9f3847b59822-kube-api-access-rwshh\") pod \"b97c2bc4-3446-4535-8566-9f3847b59822\" (UID: \"b97c2bc4-3446-4535-8566-9f3847b59822\") "
Jul 30 18:06:20.065456 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065422    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97c2bc4-3446-4535-8566-9f3847b59822-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b97c2bc4-3446-4535-8566-9f3847b59822" (UID: "b97c2bc4-3446-4535-8566-9f3847b59822"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:06:20.065516 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065483    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b97c2bc4-3446-4535-8566-9f3847b59822" (UID: "b97c2bc4-3446-4535-8566-9f3847b59822"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:06:20.065564 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.065479    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b97c2bc4-3446-4535-8566-9f3847b59822" (UID: "b97c2bc4-3446-4535-8566-9f3847b59822"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:06:20.067147 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.067118    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97c2bc4-3446-4535-8566-9f3847b59822-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b97c2bc4-3446-4535-8566-9f3847b59822" (UID: "b97c2bc4-3446-4535-8566-9f3847b59822"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:06:20.067246 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.067153    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97c2bc4-3446-4535-8566-9f3847b59822-kube-api-access-rwshh" (OuterVolumeSpecName: "kube-api-access-rwshh") pod "b97c2bc4-3446-4535-8566-9f3847b59822" (UID: "b97c2bc4-3446-4535-8566-9f3847b59822"). InnerVolumeSpecName "kube-api-access-rwshh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:06:20.166427 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.166385    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwshh\" (UniqueName: \"kubernetes.io/projected/b97c2bc4-3446-4535-8566-9f3847b59822-kube-api-access-rwshh\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:06:20.166427 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.166420    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b97c2bc4-3446-4535-8566-9f3847b59822-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:06:20.166427 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.166433    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b97c2bc4-3446-4535-8566-9f3847b59822-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:06:20.166674 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.166442    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:06:20.166674 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.166466    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b97c2bc4-3446-4535-8566-9f3847b59822-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:06:20.723101 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.723068    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"
Jul 30 18:06:20.723543 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.723077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht" event={"ID":"b97c2bc4-3446-4535-8566-9f3847b59822","Type":"ContainerDied","Data":"6ea191802e5c10c466c956245ac7951c169413963456a549e93e72018725413b"}
Jul 30 18:06:20.723543 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.723214    2569 scope.go:117] "RemoveContainer" containerID="2dde32f3ca2656d22b435fa25793d6a8927f22eaf5140bca67777797d5e910a9"
Jul 30 18:06:20.730891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.730871    2569 scope.go:117] "RemoveContainer" containerID="cb2a2e3b91e5e0fada8d9f311f7a30a2de6b96bcf6c3e73dd728f7e3848ba44e"
Jul 30 18:06:20.737963 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.737946    2569 scope.go:117] "RemoveContainer" containerID="e29ef8abf72893771902935f487d01aedee97a5678a3c6de077084e7c7c484e8"
Jul 30 18:06:20.746396 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.746372    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"]
Jul 30 18:06:20.754559 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:20.754535    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-r9mht"]
Jul 30 18:06:22.651849 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:22.651814    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" path="/var/lib/kubelet/pods/b97c2bc4-3446-4535-8566-9f3847b59822/volumes"
Jul 30 18:06:27.271130 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271043    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"]
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271312    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271325    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271338    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kube-rbac-proxy"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271343    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kube-rbac-proxy"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271350    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="storage-initializer"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271356    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="storage-initializer"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271408    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kube-rbac-proxy"
Jul 30 18:06:27.271522 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.271417    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97c2bc4-3446-4535-8566-9f3847b59822" containerName="kserve-container"
Jul 30 18:06:27.275963 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.275943    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.278183 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.278162    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 30 18:06:27.278547 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.278529    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:06:27.278613 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.278543    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:06:27.278648 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.278544    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:06:27.278648 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.278549    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:06:27.279391 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.279375    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:06:27.279474 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.279400    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 30 18:06:27.284328 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.284309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"]
Jul 30 18:06:27.324851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.324813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.324851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.324851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-proxy-tls\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.325044 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.324895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kserve-provision-location\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.325044 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.324944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk9dx\" (UniqueName: \"kubernetes.io/projected/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-api-access-rk9dx\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.325044 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.325002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-cabundle-cert\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.425601 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.425559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk9dx\" (UniqueName: \"kubernetes.io/projected/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-api-access-rk9dx\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.425784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.425617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-cabundle-cert\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.425784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.425642    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.425784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.425664    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-proxy-tls\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.425784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.425704    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kserve-provision-location\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.426137 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.426115    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kserve-provision-location\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.426487 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.426463    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.426572 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.426468    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-cabundle-cert\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.428095 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.428075    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-proxy-tls\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.435825 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.435801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk9dx\" (UniqueName: \"kubernetes.io/projected/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-api-access-rk9dx\") pod \"isvc-sklearn-predictor-7599977f6b-q99dq\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.586731 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.586641    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:27.716672 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.716623    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"]
Jul 30 18:06:27.719576 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:06:27.719548    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b81e69f_c38d_421a_97ce_bce42ebcd8c7.slice/crio-dfb62d754a3922d5b4d612d1e01e1631b98b753be654ba52c91971f724d6d939 WatchSource:0}: Error finding container dfb62d754a3922d5b4d612d1e01e1631b98b753be654ba52c91971f724d6d939: Status 404 returned error can't find the container with id dfb62d754a3922d5b4d612d1e01e1631b98b753be654ba52c91971f724d6d939
Jul 30 18:06:27.750210 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:27.750179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerStarted","Data":"dfb62d754a3922d5b4d612d1e01e1631b98b753be654ba52c91971f724d6d939"}
Jul 30 18:06:28.755295 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:28.755241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerStarted","Data":"1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1"}
Jul 30 18:06:31.768179 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:31.768144    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerID="1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1" exitCode=0
Jul 30 18:06:31.768577 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:31.768200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerDied","Data":"1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1"}
Jul 30 18:06:32.774104 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:32.774066    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerStarted","Data":"703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c"}
Jul 30 18:06:32.774466 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:32.774127    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerStarted","Data":"990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86"}
Jul 30 18:06:32.774466 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:32.774309    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:32.798168 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:32.798121    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podStartSLOduration=2.009840841 podStartE2EDuration="5.798105072s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788264231s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:06:32.796747308 +0000 UTC m=+3960.829557660" watchObservedRunningTime="2026-07-30 18:06:32.798105072 +0000 UTC m=+3960.830915409"
Jul 30 18:06:33.778167 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:33.778128    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:33.779505 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:33.779478    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:06:34.781705 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:34.781644    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:06:39.785891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:39.785859    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:06:39.786436 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:39.786361    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:06:49.786697 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:49.786656    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:06:59.786496 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:06:59.786450    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:07:09.787105 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:09.787068    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:07:19.787064 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:19.787023    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:07:29.786938 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:29.786901    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:07:39.786945 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:39.786908    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:07:43.647553 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:43.647513    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 30 18:07:53.648163 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:53.648072    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:07:57.436349 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:57.436316    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"]
Jul 30 18:07:57.436718 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:57.436578    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container" containerID="cri-o://990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86" gracePeriod=30
Jul 30 18:07:57.436718 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:57.436598    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kube-rbac-proxy" containerID="cri-o://703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c" gracePeriod=30
Jul 30 18:07:58.095845 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:58.095802    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerID="703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c" exitCode=2
Jul 30 18:07:58.096046 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:58.095883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerDied","Data":"703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c"}
Jul 30 18:07:59.782306 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:07:59.782247    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 30 18:08:01.973104 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:01.973078    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:08:02.083828 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.083732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-rbac-proxy-sar-config\") pod \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") "
Jul 30 18:08:02.083828 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.083786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-cabundle-cert\") pod \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") "
Jul 30 18:08:02.084077 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.083850    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk9dx\" (UniqueName: \"kubernetes.io/projected/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-api-access-rk9dx\") pod \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") "
Jul 30 18:08:02.084077 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.083881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kserve-provision-location\") pod \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") "
Jul 30 18:08:02.084193 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-proxy-tls\") pod \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\" (UID: \"7b81e69f-c38d-421a-97ce-bce42ebcd8c7\") "
Jul 30 18:08:02.084250 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084225    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b81e69f-c38d-421a-97ce-bce42ebcd8c7" (UID: "7b81e69f-c38d-421a-97ce-bce42ebcd8c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:08:02.084250 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084229    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b81e69f-c38d-421a-97ce-bce42ebcd8c7" (UID: "7b81e69f-c38d-421a-97ce-bce42ebcd8c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:08:02.084374 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084239    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b81e69f-c38d-421a-97ce-bce42ebcd8c7" (UID: "7b81e69f-c38d-421a-97ce-bce42ebcd8c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:08:02.084413 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084401    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:08:02.084447 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084420    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:08:02.084447 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.084433    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:08:02.086029 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.086007    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b81e69f-c38d-421a-97ce-bce42ebcd8c7" (UID: "7b81e69f-c38d-421a-97ce-bce42ebcd8c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:08:02.086136 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.086119    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-api-access-rk9dx" (OuterVolumeSpecName: "kube-api-access-rk9dx") pod "7b81e69f-c38d-421a-97ce-bce42ebcd8c7" (UID: "7b81e69f-c38d-421a-97ce-bce42ebcd8c7"). InnerVolumeSpecName "kube-api-access-rk9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:08:02.111852 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.111817    2569 generic.go:356] "Generic (PLEG): container finished" podID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerID="990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86" exitCode=0
Jul 30 18:08:02.111993 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.111885    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"
Jul 30 18:08:02.111993 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.111885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerDied","Data":"990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86"}
Jul 30 18:08:02.111993 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.111945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq" event={"ID":"7b81e69f-c38d-421a-97ce-bce42ebcd8c7","Type":"ContainerDied","Data":"dfb62d754a3922d5b4d612d1e01e1631b98b753be654ba52c91971f724d6d939"}
Jul 30 18:08:02.111993 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.111972    2569 scope.go:117] "RemoveContainer" containerID="703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c"
Jul 30 18:08:02.120156 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.120136    2569 scope.go:117] "RemoveContainer" containerID="990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86"
Jul 30 18:08:02.127123 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.127106    2569 scope.go:117] "RemoveContainer" containerID="1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1"
Jul 30 18:08:02.134132 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.133980    2569 scope.go:117] "RemoveContainer" containerID="703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c"
Jul 30 18:08:02.134476 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:08:02.134451    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c\": container with ID starting with 703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c not found: ID does not exist" containerID="703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c"
Jul 30 18:08:02.134594 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.134480    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c"} err="failed to get container status \"703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c\": rpc error: code = NotFound desc = could not find container \"703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c\": container with ID starting with 703966bedece0223bd101c8cf20c889562839f337ba15c9f3d9af6300698bb2c not found: ID does not exist"
Jul 30 18:08:02.134594 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.134500    2569 scope.go:117] "RemoveContainer" containerID="990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86"
Jul 30 18:08:02.134782 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:08:02.134763    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86\": container with ID starting with 990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86 not found: ID does not exist" containerID="990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86"
Jul 30 18:08:02.134869 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.134787    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86"} err="failed to get container status \"990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86\": rpc error: code = NotFound desc = could not find container \"990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86\": container with ID starting with 990e19ee1062453062e7d1292cfa3c0ef07075f560f2649829de7217036fcb86 not found: ID does not exist"
Jul 30 18:08:02.134869 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.134808    2569 scope.go:117] "RemoveContainer" containerID="1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1"
Jul 30 18:08:02.135070 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:08:02.135047    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1\": container with ID starting with 1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1 not found: ID does not exist" containerID="1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1"
Jul 30 18:08:02.135130 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.135076    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1"} err="failed to get container status \"1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1\": rpc error: code = NotFound desc = could not find container \"1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1\": container with ID starting with 1eddbf4696e366bd52d3cce0d7e69fe76840538cabacac081d4d159f4e72acf1 not found: ID does not exist"
Jul 30 18:08:02.136500 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.136477    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"]
Jul 30 18:08:02.141071 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.141050    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7599977f6b-q99dq"]
Jul 30 18:08:02.184995 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.184963    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk9dx\" (UniqueName: \"kubernetes.io/projected/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-kube-api-access-rk9dx\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:08:02.184995 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.184991    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b81e69f-c38d-421a-97ce-bce42ebcd8c7-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:08:02.651816 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:08:02.651784    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" path="/var/lib/kubelet/pods/7b81e69f-c38d-421a-97ce-bce42ebcd8c7/volumes"
Jul 30 18:09:07.855215 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855173    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"]
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855583    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kube-rbac-proxy"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855604    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kube-rbac-proxy"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855629    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855639    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855652    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="storage-initializer"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855662    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="storage-initializer"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855724    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kube-rbac-proxy"
Jul 30 18:09:07.855774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.855736    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b81e69f-c38d-421a-97ce-bce42ebcd8c7" containerName="kserve-container"
Jul 30 18:09:07.859125 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.859101    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:07.861756 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.861727    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 30 18:09:07.861946 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.861922    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:09:07.862050 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.861977    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 30 18:09:07.862886 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.862865    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:09:07.862994 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.862924    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:09:07.862994 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.862952    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:09:07.862994 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.862967    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:09:07.872202 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.872182    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"]
Jul 30 18:09:07.976908 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.976875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:07.976908 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.976913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:07.977123 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.976943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:07.977123 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.976971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltw5m\" (UniqueName: \"kubernetes.io/projected/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-api-access-ltw5m\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:07.977123 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:07.977018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.077809 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.077770    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltw5m\" (UniqueName: \"kubernetes.io/projected/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-api-access-ltw5m\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078018 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.077832    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078018 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.077880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078018 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.077903    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078018 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.077933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078236 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:09:08.078076    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 30 18:09:08.078236 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:09:08.078139    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls podName:1bdf4579-25a4-48f2-ba8a-2ce67e524f6a nodeName:}" failed. No retries permitted until 2026-07-30 18:09:08.578118786 +0000 UTC m=+4116.610929103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls") pod "isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" (UID: "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 30 18:09:08.078465 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.078340    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078614 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.078594    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.078752 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.078731    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.089423 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.089388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltw5m\" (UniqueName: \"kubernetes.io/projected/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-api-access-ltw5m\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.582847 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.582807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.585060 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.585035    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-dc457f6d6-vtphp\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.768891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.768861    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:08.897986 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:08.897860    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"]
Jul 30 18:09:08.900525 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:09:08.900498    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdf4579_25a4_48f2_ba8a_2ce67e524f6a.slice/crio-a00b121e019df4ffb2fe4e68686c565495442d9b35c2068e8975142adaddf72e WatchSource:0}: Error finding container a00b121e019df4ffb2fe4e68686c565495442d9b35c2068e8975142adaddf72e: Status 404 returned error can't find the container with id a00b121e019df4ffb2fe4e68686c565495442d9b35c2068e8975142adaddf72e
Jul 30 18:09:09.355588 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:09.355547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerStarted","Data":"607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720"}
Jul 30 18:09:09.355774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:09.355638    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerStarted","Data":"a00b121e019df4ffb2fe4e68686c565495442d9b35c2068e8975142adaddf72e"}
Jul 30 18:09:15.378764 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:15.378729    2569 generic.go:356] "Generic (PLEG): container finished" podID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerID="607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720" exitCode=0
Jul 30 18:09:15.379137 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:15.378799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerDied","Data":"607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720"}
Jul 30 18:09:16.385493 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:16.385452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerStarted","Data":"797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7"}
Jul 30 18:09:16.385905 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:16.385552    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerStarted","Data":"518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97"}
Jul 30 18:09:16.385905 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:16.385803    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:16.407990 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:16.407941    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" podStartSLOduration=3.236856232 podStartE2EDuration="9.407926073s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.171069841s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:09:16.407340181 +0000 UTC m=+4124.440150517" watchObservedRunningTime="2026-07-30 18:09:16.407926073 +0000 UTC m=+4124.440736411"
Jul 30 18:09:17.389897 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:17.389863    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:17.390873 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:17.390849    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 30 18:09:18.394114 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:18.394065    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 30 18:09:23.398951 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:23.398917    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:23.399654 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:23.399627    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 30 18:09:33.400418 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:33.400390    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:39.606233 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:39.606200    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-dc457f6d6-vtphp_1bdf4579-25a4-48f2-ba8a-2ce67e524f6a/kserve-container/0.log"
Jul 30 18:09:39.777760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:39.777727    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"]
Jul 30 18:09:39.778021 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:39.777996    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container" containerID="cri-o://518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97" gracePeriod=30
Jul 30 18:09:39.778079 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:39.778027    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kube-rbac-proxy" containerID="cri-o://797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7" gracePeriod=30
Jul 30 18:09:40.482389 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:40.482350    2569 generic.go:356] "Generic (PLEG): container finished" podID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerID="797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7" exitCode=2
Jul 30 18:09:40.482559 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:40.482405    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerDied","Data":"797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7"}
Jul 30 18:09:41.025531 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.025507    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:41.125761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.125732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-rbac-proxy-sar-config\") pod \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") "
Jul 30 18:09:41.125761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.125765    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltw5m\" (UniqueName: \"kubernetes.io/projected/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-api-access-ltw5m\") pod \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") "
Jul 30 18:09:41.126020 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.125794    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-cabundle-cert\") pod \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") "
Jul 30 18:09:41.126020 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.125816    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls\") pod \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") "
Jul 30 18:09:41.126020 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.125850    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kserve-provision-location\") pod \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\" (UID: \"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a\") "
Jul 30 18:09:41.126195 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.126167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" (UID: "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:09:41.126276 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.126170    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" (UID: "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:09:41.127993 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.127936    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" (UID: "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:09:41.128111 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.128084    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-api-access-ltw5m" (OuterVolumeSpecName: "kube-api-access-ltw5m") pod "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" (UID: "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a"). InnerVolumeSpecName "kube-api-access-ltw5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:09:41.152501 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.152460    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" (UID: "1bdf4579-25a4-48f2-ba8a-2ce67e524f6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:09:41.226315 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.226282    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:09:41.226315 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.226310    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltw5m\" (UniqueName: \"kubernetes.io/projected/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kube-api-access-ltw5m\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:09:41.226315 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.226320    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:09:41.226542 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.226330    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:09:41.226542 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.226338    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:09:41.488322 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.488283    2569 generic.go:356] "Generic (PLEG): container finished" podID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerID="518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97" exitCode=0
Jul 30 18:09:41.488495 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.488336    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"
Jul 30 18:09:41.488495 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.488362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerDied","Data":"518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97"}
Jul 30 18:09:41.488495 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.488434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp" event={"ID":"1bdf4579-25a4-48f2-ba8a-2ce67e524f6a","Type":"ContainerDied","Data":"a00b121e019df4ffb2fe4e68686c565495442d9b35c2068e8975142adaddf72e"}
Jul 30 18:09:41.488495 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.488475    2569 scope.go:117] "RemoveContainer" containerID="797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7"
Jul 30 18:09:41.496718 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.496699    2569 scope.go:117] "RemoveContainer" containerID="518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97"
Jul 30 18:09:41.503858 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.503838    2569 scope.go:117] "RemoveContainer" containerID="607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720"
Jul 30 18:09:41.511302 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.511282    2569 scope.go:117] "RemoveContainer" containerID="797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7"
Jul 30 18:09:41.511549 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:09:41.511529    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7\": container with ID starting with 797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7 not found: ID does not exist" containerID="797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7"
Jul 30 18:09:41.511623 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.511562    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7"} err="failed to get container status \"797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7\": rpc error: code = NotFound desc = could not find container \"797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7\": container with ID starting with 797eafaebcee046c944ffdaca14ebb0851f0f599ba725b9be59626c61a6974d7 not found: ID does not exist"
Jul 30 18:09:41.511623 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.511588    2569 scope.go:117] "RemoveContainer" containerID="518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97"
Jul 30 18:09:41.511839 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:09:41.511821    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97\": container with ID starting with 518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97 not found: ID does not exist" containerID="518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97"
Jul 30 18:09:41.511909 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.511846    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97"} err="failed to get container status \"518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97\": rpc error: code = NotFound desc = could not find container \"518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97\": container with ID starting with 518eb6347cc3cac8edb2aecc8c650419008a43f9150de720ff45bd848e960e97 not found: ID does not exist"
Jul 30 18:09:41.511909 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.511860    2569 scope.go:117] "RemoveContainer" containerID="607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720"
Jul 30 18:09:41.512084 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:09:41.512066    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720\": container with ID starting with 607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720 not found: ID does not exist" containerID="607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720"
Jul 30 18:09:41.512137 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.512093    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720"} err="failed to get container status \"607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720\": rpc error: code = NotFound desc = could not find container \"607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720\": container with ID starting with 607d82823f7d6bb6add900571cd15349e05c250034e1edab9949353009a1e720 not found: ID does not exist"
Jul 30 18:09:41.516991 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.516969    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"]
Jul 30 18:09:41.519578 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:41.519557    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-dc457f6d6-vtphp"]
Jul 30 18:09:42.651230 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:09:42.651201    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" path="/var/lib/kubelet/pods/1bdf4579-25a4-48f2-ba8a-2ce67e524f6a/volumes"
Jul 30 18:10:50.216202 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216160    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"]
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216460    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216474    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216484    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kube-rbac-proxy"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216490    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kube-rbac-proxy"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216498    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="storage-initializer"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216504    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="storage-initializer"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216551    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kserve-container"
Jul 30 18:10:50.216700 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.216560    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdf4579-25a4-48f2-ba8a-2ce67e524f6a" containerName="kube-rbac-proxy"
Jul 30 18:10:50.225601 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.225559    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.228790 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.228762    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:10:50.228952 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.228789    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 30 18:10:50.228952 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.228805    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:10:50.228952 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.228896    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 30 18:10:50.229122 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.229053    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:10:50.229898 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.229870    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:10:50.230092 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.229871    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:10:50.231644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.231623    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"]
Jul 30 18:10:50.244807 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.244786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrmx\" (UniqueName: \"kubernetes.io/projected/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-api-access-fqrmx\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.244913 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.244835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.244913 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.244862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.244913 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.244894    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.245033 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.244921    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.345230 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.345183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.345230 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.345230    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.345527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.345368    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.345527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.345424    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.345527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.345464    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrmx\" (UniqueName: \"kubernetes.io/projected/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-api-access-fqrmx\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.345700 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:10:50.345555    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 30 18:10:50.345700 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:10:50.345647    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls podName:e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb nodeName:}" failed. No retries permitted until 2026-07-30 18:10:50.845624618 +0000 UTC m=+4218.878434936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls") pod "isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" (UID: "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 30 18:10:50.345804 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.345688    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.346023 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.346005    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.346023 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.346016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.355465 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.355442    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrmx\" (UniqueName: \"kubernetes.io/projected/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-api-access-fqrmx\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.849356 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.849325    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:50.851601 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:50.851577    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:51.138293 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:51.138179    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:51.267094 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:51.267068    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"]
Jul 30 18:10:51.269470 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:10:51.269440    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f03fe4_8c24_428b_9b9f_6b0204cbb4bb.slice/crio-e8da0cd1454eb5cc604020feea76e31ce5163c0d1a29538945e65a05b1cf9ae7 WatchSource:0}: Error finding container e8da0cd1454eb5cc604020feea76e31ce5163c0d1a29538945e65a05b1cf9ae7: Status 404 returned error can't find the container with id e8da0cd1454eb5cc604020feea76e31ce5163c0d1a29538945e65a05b1cf9ae7
Jul 30 18:10:51.748974 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:51.748936    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerStarted","Data":"1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705"}
Jul 30 18:10:51.749173 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:51.749008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerStarted","Data":"e8da0cd1454eb5cc604020feea76e31ce5163c0d1a29538945e65a05b1cf9ae7"}
Jul 30 18:10:55.765332 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:55.765293    2569 generic.go:356] "Generic (PLEG): container finished" podID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerID="1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705" exitCode=0
Jul 30 18:10:55.765742 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:55.765369    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerDied","Data":"1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705"}
Jul 30 18:10:56.770774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:56.770738    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerStarted","Data":"7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a"}
Jul 30 18:10:56.771138 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:56.770800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerStarted","Data":"bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e"}
Jul 30 18:10:56.771138 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:56.771001    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:56.793783 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:56.793733    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podStartSLOduration=3.248725861 podStartE2EDuration="6.793717213s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.544991352s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:10:56.793174224 +0000 UTC m=+4224.825984575" watchObservedRunningTime="2026-07-30 18:10:56.793717213 +0000 UTC m=+4224.826527548"
Jul 30 18:10:57.774750 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:57.774717    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:10:57.775879 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:57.775854    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:10:58.778721 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:10:58.778682    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:11:03.783171 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:03.783138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:11:03.783767 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:03.783740    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:11:13.784119 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:13.784071    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:11:23.783880 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:23.783838    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:11:33.783971 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:33.783931    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:11:43.783996 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:43.783957    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:11:53.783912 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:11:53.783874    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:12:03.784497 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:03.784452    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:12:08.647980 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:08.647940    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 30 18:12:18.651436 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:18.651406    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:12:20.425010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:20.424975    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"]
Jul 30 18:12:20.425414 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:20.425212    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container" containerID="cri-o://bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e" gracePeriod=30
Jul 30 18:12:20.425414 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:20.425247    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kube-rbac-proxy" containerID="cri-o://7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a" gracePeriod=30
Jul 30 18:12:21.083149 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:21.083116    2569 generic.go:356] "Generic (PLEG): container finished" podID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerID="7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a" exitCode=2
Jul 30 18:12:21.083383 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:21.083179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerDied","Data":"7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a"}
Jul 30 18:12:23.779904 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:23.779805    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 30 18:12:25.074634 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.074606    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:12:25.101526 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.101494    2569 generic.go:356] "Generic (PLEG): container finished" podID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerID="bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e" exitCode=0
Jul 30 18:12:25.101706 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.101551    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"
Jul 30 18:12:25.101706 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.101577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerDied","Data":"bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e"}
Jul 30 18:12:25.101706 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.101653    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b" event={"ID":"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb","Type":"ContainerDied","Data":"e8da0cd1454eb5cc604020feea76e31ce5163c0d1a29538945e65a05b1cf9ae7"}
Jul 30 18:12:25.101706 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.101696    2569 scope.go:117] "RemoveContainer" containerID="7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a"
Jul 30 18:12:25.112502 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.112472    2569 scope.go:117] "RemoveContainer" containerID="bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e"
Jul 30 18:12:25.119548 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.119523    2569 scope.go:117] "RemoveContainer" containerID="1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705"
Jul 30 18:12:25.126344 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.126327    2569 scope.go:117] "RemoveContainer" containerID="7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a"
Jul 30 18:12:25.126609 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:12:25.126591    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a\": container with ID starting with 7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a not found: ID does not exist" containerID="7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a"
Jul 30 18:12:25.126677 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.126623    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a"} err="failed to get container status \"7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a\": rpc error: code = NotFound desc = could not find container \"7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a\": container with ID starting with 7e10f8c0bb383e6e1b8be1a1e8e3f2928898bc90adfafabd9a96fa48556c5b2a not found: ID does not exist"
Jul 30 18:12:25.126677 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.126642    2569 scope.go:117] "RemoveContainer" containerID="bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e"
Jul 30 18:12:25.126876 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:12:25.126859    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e\": container with ID starting with bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e not found: ID does not exist" containerID="bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e"
Jul 30 18:12:25.126921 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.126880    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e"} err="failed to get container status \"bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e\": rpc error: code = NotFound desc = could not find container \"bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e\": container with ID starting with bdd495461644ec8c74b4782e4c69014a6eeb203e190cf777f5f220122fa2d92e not found: ID does not exist"
Jul 30 18:12:25.126921 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.126893    2569 scope.go:117] "RemoveContainer" containerID="1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705"
Jul 30 18:12:25.127122 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:12:25.127109    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705\": container with ID starting with 1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705 not found: ID does not exist" containerID="1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705"
Jul 30 18:12:25.127168 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.127123    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705"} err="failed to get container status \"1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705\": rpc error: code = NotFound desc = could not find container \"1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705\": container with ID starting with 1e578c45f5491d501ded8297e1ccd5230493ef253b80b0ac9d259817e0312705 not found: ID does not exist"
Jul 30 18:12:25.153193 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153167    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqrmx\" (UniqueName: \"kubernetes.io/projected/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-api-access-fqrmx\") pod \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") "
Jul 30 18:12:25.153348 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153203    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kserve-provision-location\") pod \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") "
Jul 30 18:12:25.153348 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153229    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-cabundle-cert\") pod \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") "
Jul 30 18:12:25.153348 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153311    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls\") pod \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") "
Jul 30 18:12:25.153526 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153361    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-rbac-proxy-sar-config\") pod \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\" (UID: \"e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb\") "
Jul 30 18:12:25.153582 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153534    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" (UID: "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:12:25.153631 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153600    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" (UID: "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:12:25.153769 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.153746    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" (UID: "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:12:25.155288 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.155249    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-api-access-fqrmx" (OuterVolumeSpecName: "kube-api-access-fqrmx") pod "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" (UID: "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb"). InnerVolumeSpecName "kube-api-access-fqrmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:12:25.155380 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.155312    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" (UID: "e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:12:25.253881 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.253849    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqrmx\" (UniqueName: \"kubernetes.io/projected/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-api-access-fqrmx\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:12:25.253881 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.253879    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:12:25.254085 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.253889    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:12:25.254085 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.253900    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:12:25.254085 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.253908    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:12:25.425804 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.425778    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"]
Jul 30 18:12:25.433604 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.433578    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-8c7fcfb4b-h5n2b"]
Jul 30 18:12:25.609038 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.608958    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"]
Jul 30 18:12:25.609225 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609213    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kube-rbac-proxy"
Jul 30 18:12:25.609284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609226    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kube-rbac-proxy"
Jul 30 18:12:25.609284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609246    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="storage-initializer"
Jul 30 18:12:25.609284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609264    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="storage-initializer"
Jul 30 18:12:25.609284 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609277    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container"
Jul 30 18:12:25.609419 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609286    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container"
Jul 30 18:12:25.609419 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609326    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kserve-container"
Jul 30 18:12:25.609419 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.609335    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" containerName="kube-rbac-proxy"
Jul 30 18:12:25.613718 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.613697    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.616363 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616341    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:12:25.616497 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:12:25.616497 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616460    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:12:25.616497 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616467    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:12:25.616681 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616595    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 30 18:12:25.616787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616736    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 30 18:12:25.616884 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.616826    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:12:25.622884 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.622863    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"]
Jul 30 18:12:25.656023 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.655993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7s2\" (UniqueName: \"kubernetes.io/projected/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-api-access-4g7s2\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.656198 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.656032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.656198 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.656051    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.656198 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.656102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.656344 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.656199    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.757381 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.757346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.757581 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.757394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7s2\" (UniqueName: \"kubernetes.io/projected/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-api-access-4g7s2\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.757581 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.757416    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.757581 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.757437    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.757581 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.757471    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.757832 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:12:25.757693    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 30 18:12:25.757832 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:12:25.757772    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls podName:5b9c0bae-1bb2-4340-ac51-291b49fcfd98 nodeName:}" failed. No retries permitted until 2026-07-30 18:12:26.257750236 +0000 UTC m=+4314.290560560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" (UID: "5b9c0bae-1bb2-4340-ac51-291b49fcfd98") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 30 18:12:25.757952 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.757905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.758160 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.758137    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.758227 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.758159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:25.767149 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:25.767129    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7s2\" (UniqueName: \"kubernetes.io/projected/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-api-access-4g7s2\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:26.260985 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:26.260925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:26.263458 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:26.263435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:26.524792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:26.524690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:26.651536 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:26.651506    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb" path="/var/lib/kubelet/pods/e4f03fe4-8c24-428b-9b9f-6b0204cbb4bb/volumes"
Jul 30 18:12:26.654888 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:26.654848    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"]
Jul 30 18:12:26.656996 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:12:26.656968    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9c0bae_1bb2_4340_ac51_291b49fcfd98.slice/crio-e80834cff39a711b6998b36da4f4cb4486f7681a648b5f6db24d1edd44aa38db WatchSource:0}: Error finding container e80834cff39a711b6998b36da4f4cb4486f7681a648b5f6db24d1edd44aa38db: Status 404 returned error can't find the container with id e80834cff39a711b6998b36da4f4cb4486f7681a648b5f6db24d1edd44aa38db
Jul 30 18:12:27.111216 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:27.111185    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerStarted","Data":"4c7285fba12abf9df2648b5a8b285487d934118879b80895ff6962f234523508"}
Jul 30 18:12:27.111401 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:27.111238    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerStarted","Data":"e80834cff39a711b6998b36da4f4cb4486f7681a648b5f6db24d1edd44aa38db"}
Jul 30 18:12:31.127348 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:31.127247    2569 generic.go:356] "Generic (PLEG): container finished" podID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerID="4c7285fba12abf9df2648b5a8b285487d934118879b80895ff6962f234523508" exitCode=0
Jul 30 18:12:31.127348 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:31.127326    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerDied","Data":"4c7285fba12abf9df2648b5a8b285487d934118879b80895ff6962f234523508"}
Jul 30 18:12:32.133391 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:32.133355    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerStarted","Data":"61c456a7932ce8b7ab3dbf3e59b44e41a8a06cbbb11709a31ce0a44d76adb836"}
Jul 30 18:12:32.133775 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:32.133416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerStarted","Data":"ea2c699c0243501640655dec3924fbc0e1fdca98886294443712c74c4812ed39"}
Jul 30 18:12:32.133775 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:32.133681    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:32.133847 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:32.133816    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:32.134924 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:32.134901    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:12:32.155378 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:32.155332    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podStartSLOduration=3.175849528 podStartE2EDuration="7.155317308s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.97946778s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:12:32.1533833 +0000 UTC m=+4320.186193637" watchObservedRunningTime="2026-07-30 18:12:32.155317308 +0000 UTC m=+4320.188127644"
Jul 30 18:12:33.136955 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:33.136918    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:12:38.141029 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:38.140996    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:12:38.141658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:38.141632    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:12:48.141636 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:48.141587    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:12:58.141975 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:12:58.141932    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:13:08.142401 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:08.142359    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:13:18.142379 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:18.142338    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:13:28.141797 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:28.141752    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:13:38.142505 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:38.142464    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:13:48.142334 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:48.142302    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:13:55.700964 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:55.700890    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"]
Jul 30 18:13:55.701477 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:55.701138    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" containerID="cri-o://ea2c699c0243501640655dec3924fbc0e1fdca98886294443712c74c4812ed39" gracePeriod=30
Jul 30 18:13:55.701477 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:55.701175    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kube-rbac-proxy" containerID="cri-o://61c456a7932ce8b7ab3dbf3e59b44e41a8a06cbbb11709a31ce0a44d76adb836" gracePeriod=30
Jul 30 18:13:56.446270 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:56.446227    2569 generic.go:356] "Generic (PLEG): container finished" podID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerID="61c456a7932ce8b7ab3dbf3e59b44e41a8a06cbbb11709a31ce0a44d76adb836" exitCode=2
Jul 30 18:13:56.446437 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:56.446287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerDied","Data":"61c456a7932ce8b7ab3dbf3e59b44e41a8a06cbbb11709a31ce0a44d76adb836"}
Jul 30 18:13:58.138056 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:58.138010    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jul 30 18:13:58.142355 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:13:58.142332    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 30 18:14:00.462948 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.462917    2569 generic.go:356] "Generic (PLEG): container finished" podID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerID="ea2c699c0243501640655dec3924fbc0e1fdca98886294443712c74c4812ed39" exitCode=0
Jul 30 18:14:00.463338 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.462999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerDied","Data":"ea2c699c0243501640655dec3924fbc0e1fdca98886294443712c74c4812ed39"}
Jul 30 18:14:00.543952 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.543927    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:14:00.697972 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.697944    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kserve-provision-location\") pod \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") "
Jul 30 18:14:00.698375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.697978    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-rbac-proxy-sar-config\") pod \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") "
Jul 30 18:14:00.698375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.697997    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-cabundle-cert\") pod \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") "
Jul 30 18:14:00.698375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.698029    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls\") pod \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") "
Jul 30 18:14:00.698375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.698099    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g7s2\" (UniqueName: \"kubernetes.io/projected/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-api-access-4g7s2\") pod \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\" (UID: \"5b9c0bae-1bb2-4340-ac51-291b49fcfd98\") "
Jul 30 18:14:00.698375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.698328    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b9c0bae-1bb2-4340-ac51-291b49fcfd98" (UID: "5b9c0bae-1bb2-4340-ac51-291b49fcfd98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:14:00.698375 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.698371    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5b9c0bae-1bb2-4340-ac51-291b49fcfd98" (UID: "5b9c0bae-1bb2-4340-ac51-291b49fcfd98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:14:00.698801 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.698383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b9c0bae-1bb2-4340-ac51-291b49fcfd98" (UID: "5b9c0bae-1bb2-4340-ac51-291b49fcfd98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:14:00.700191 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.700163    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b9c0bae-1bb2-4340-ac51-291b49fcfd98" (UID: "5b9c0bae-1bb2-4340-ac51-291b49fcfd98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:14:00.700364 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.700343    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-api-access-4g7s2" (OuterVolumeSpecName: "kube-api-access-4g7s2") pod "5b9c0bae-1bb2-4340-ac51-291b49fcfd98" (UID: "5b9c0bae-1bb2-4340-ac51-291b49fcfd98"). InnerVolumeSpecName "kube-api-access-4g7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:14:00.799182 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.799151    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:14:00.799182 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.799179    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:14:00.799182 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.799190    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:14:00.799418 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.799200    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:14:00.799418 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.799209    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7s2\" (UniqueName: \"kubernetes.io/projected/5b9c0bae-1bb2-4340-ac51-291b49fcfd98-kube-api-access-4g7s2\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:14:00.873573 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873542    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"]
Jul 30 18:14:00.873813 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873801    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container"
Jul 30 18:14:00.873858 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873814    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container"
Jul 30 18:14:00.873858 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873831    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kube-rbac-proxy"
Jul 30 18:14:00.873858 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873837    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kube-rbac-proxy"
Jul 30 18:14:00.873858 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873845    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="storage-initializer"
Jul 30 18:14:00.873858 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873851    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="storage-initializer"
Jul 30 18:14:00.874011 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873892    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kserve-container"
Jul 30 18:14:00.874011 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.873900    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" containerName="kube-rbac-proxy"
Jul 30 18:14:00.876968 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.876949    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:00.879342 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.879321    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 30 18:14:00.879471 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.879385    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 30 18:14:00.887598 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.887577    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"]
Jul 30 18:14:01.000049 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:00.999942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.000049 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.000000    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ft68\" (UniqueName: \"kubernetes.io/projected/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-api-access-8ft68\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.000250 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.000098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.000250 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.000136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.000250 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.000171    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101245 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101191    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101245 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101480 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101480 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101480 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ft68\" (UniqueName: \"kubernetes.io/projected/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-api-access-8ft68\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101611 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:14:01.101481    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 30 18:14:01.101611 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:14:01.101566    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls podName:3dae9f65-69d5-4400-bc2d-41d1e3f4b270 nodeName:}" failed. No retries permitted until 2026-07-30 18:14:01.601544657 +0000 UTC m=+4409.634354976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-sbh5s" (UID: "3dae9f65-69d5-4400-bc2d-41d1e3f4b270") : secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 30 18:14:01.101788 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101958 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101941    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.101995 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.101950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.113917 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.113888    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ft68\" (UniqueName: \"kubernetes.io/projected/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-api-access-8ft68\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.468156 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.468125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn" event={"ID":"5b9c0bae-1bb2-4340-ac51-291b49fcfd98","Type":"ContainerDied","Data":"e80834cff39a711b6998b36da4f4cb4486f7681a648b5f6db24d1edd44aa38db"}
Jul 30 18:14:01.468156 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.468144    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"
Jul 30 18:14:01.468770 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.468197    2569 scope.go:117] "RemoveContainer" containerID="61c456a7932ce8b7ab3dbf3e59b44e41a8a06cbbb11709a31ce0a44d76adb836"
Jul 30 18:14:01.476900 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.476877    2569 scope.go:117] "RemoveContainer" containerID="ea2c699c0243501640655dec3924fbc0e1fdca98886294443712c74c4812ed39"
Jul 30 18:14:01.484013 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.483994    2569 scope.go:117] "RemoveContainer" containerID="4c7285fba12abf9df2648b5a8b285487d934118879b80895ff6962f234523508"
Jul 30 18:14:01.492306 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.492286    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"]
Jul 30 18:14:01.496322 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.496302    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-74df6969cf-9vqnn"]
Jul 30 18:14:01.606118 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.606077    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.608523 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.608504    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-sbh5s\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.787608 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.787512    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:01.916177 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:01.916153    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"]
Jul 30 18:14:01.918592 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:14:01.918562    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dae9f65_69d5_4400_bc2d_41d1e3f4b270.slice/crio-d4c082bee03fafb1ec7efdfce01fd751da06d0ba02047fe10601e8b25950c6c9 WatchSource:0}: Error finding container d4c082bee03fafb1ec7efdfce01fd751da06d0ba02047fe10601e8b25950c6c9: Status 404 returned error can't find the container with id d4c082bee03fafb1ec7efdfce01fd751da06d0ba02047fe10601e8b25950c6c9
Jul 30 18:14:02.473566 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:02.473529    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerStarted","Data":"323f36801fff5a157ba53655e36bb76e5ccc715fdf7eb6371d33ea13ea1259d4"}
Jul 30 18:14:02.474010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:02.473593    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerStarted","Data":"d4c082bee03fafb1ec7efdfce01fd751da06d0ba02047fe10601e8b25950c6c9"}
Jul 30 18:14:02.651159 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:02.651127    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9c0bae-1bb2-4340-ac51-291b49fcfd98" path="/var/lib/kubelet/pods/5b9c0bae-1bb2-4340-ac51-291b49fcfd98/volumes"
Jul 30 18:14:07.495135 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:07.495103    2569 generic.go:356] "Generic (PLEG): container finished" podID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerID="323f36801fff5a157ba53655e36bb76e5ccc715fdf7eb6371d33ea13ea1259d4" exitCode=0
Jul 30 18:14:07.495548 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:07.495162    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerDied","Data":"323f36801fff5a157ba53655e36bb76e5ccc715fdf7eb6371d33ea13ea1259d4"}
Jul 30 18:14:07.496311 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:07.496294    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 18:14:11.515999 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:11.515956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerStarted","Data":"09a637818f40e8d0079d4939d0128c2983439801ef25c44b98539226e02210bd"}
Jul 30 18:14:11.516419 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:11.516071    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerStarted","Data":"81f28272cb7b6b391cc47d9a7fc16bf014a3c9c986fbc7c87d112a2e2110f0e5"}
Jul 30 18:14:11.516419 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:11.516388    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:11.516573 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:11.516554    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:11.517790 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:11.517761    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 30 18:14:11.539436 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:11.539350    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podStartSLOduration=2.977931192 podStartE2EDuration="11.539335952s" totalImagesPullingTime="3.792583823s" totalInitContainerRuntime="4.768820937s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:14:11.53793723 +0000 UTC m=+4419.570747582" watchObservedRunningTime="2026-07-30 18:14:11.539335952 +0000 UTC m=+4419.572146288"
Jul 30 18:14:12.520820 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:12.520779    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 30 18:14:17.525995 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:17.525966    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:17.526486 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:17.526459    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 30 18:14:27.527489 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:27.527460    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:41.671463 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:41.671426    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"]
Jul 30 18:14:41.671891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:41.671668    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container" containerID="cri-o://81f28272cb7b6b391cc47d9a7fc16bf014a3c9c986fbc7c87d112a2e2110f0e5" gracePeriod=30
Jul 30 18:14:41.671891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:41.671741    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" containerID="cri-o://09a637818f40e8d0079d4939d0128c2983439801ef25c44b98539226e02210bd" gracePeriod=30
Jul 30 18:14:42.521964 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:42.521916    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 30 18:14:42.643430 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:42.643397    2569 generic.go:356] "Generic (PLEG): container finished" podID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerID="09a637818f40e8d0079d4939d0128c2983439801ef25c44b98539226e02210bd" exitCode=2
Jul 30 18:14:42.643593 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:42.643443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerDied","Data":"09a637818f40e8d0079d4939d0128c2983439801ef25c44b98539226e02210bd"}
Jul 30 18:14:46.820351 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.820312    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"]
Jul 30 18:14:46.822942 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.822921    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.825463 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.825441    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 30 18:14:46.825605 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.825585    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 30 18:14:46.834106 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.834086    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"]
Jul 30 18:14:46.860483 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.860457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9115da-ef1d-4f92-a84c-dbf9da656287-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.860579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.860487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9115da-ef1d-4f92-a84c-dbf9da656287-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.860579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.860553    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.860658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.860597    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.860658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.860625    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qcm5\" (UniqueName: \"kubernetes.io/projected/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-api-access-5qcm5\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.961549 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.961519    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.961760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.961559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.961760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.961583    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qcm5\" (UniqueName: \"kubernetes.io/projected/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-api-access-5qcm5\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.961760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.961615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9115da-ef1d-4f92-a84c-dbf9da656287-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.961760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.961633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9115da-ef1d-4f92-a84c-dbf9da656287-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.962135 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.962108    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9115da-ef1d-4f92-a84c-dbf9da656287-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.962321 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.962300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.962321 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.962311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.964045 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.964028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9115da-ef1d-4f92-a84c-dbf9da656287-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:46.971021 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:46.971001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qcm5\" (UniqueName: \"kubernetes.io/projected/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-api-access-5qcm5\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-k67g2\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:47.132775 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:47.132690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:47.263138 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:47.263104    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"]
Jul 30 18:14:47.266659 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:14:47.266626    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9115da_ef1d_4f92_a84c_dbf9da656287.slice/crio-a236aca8f55a783f321a5bfddfdcbcddba68814e5c3272ae9f5eeaec8f597693 WatchSource:0}: Error finding container a236aca8f55a783f321a5bfddfdcbcddba68814e5c3272ae9f5eeaec8f597693: Status 404 returned error can't find the container with id a236aca8f55a783f321a5bfddfdcbcddba68814e5c3272ae9f5eeaec8f597693
Jul 30 18:14:47.521619 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:47.521574    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 30 18:14:47.662610 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:47.662569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerStarted","Data":"231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2"}
Jul 30 18:14:47.662816 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:47.662625    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerStarted","Data":"a236aca8f55a783f321a5bfddfdcbcddba68814e5c3272ae9f5eeaec8f597693"}
Jul 30 18:14:52.521330 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:52.521287    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 30 18:14:52.521733 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:52.521407    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:14:52.686646 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:52.686609    2569 generic.go:356] "Generic (PLEG): container finished" podID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerID="231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2" exitCode=0
Jul 30 18:14:52.686646 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:52.686635    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerDied","Data":"231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2"}
Jul 30 18:14:53.691827 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:53.691789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerStarted","Data":"5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18"}
Jul 30 18:14:53.692231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:53.691854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerStarted","Data":"a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e"}
Jul 30 18:14:53.692231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:53.692047    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:53.712521 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:53.712475    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podStartSLOduration=3.274046851 podStartE2EDuration="7.712460494s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.438413643s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:14:53.711352876 +0000 UTC m=+4461.744163214" watchObservedRunningTime="2026-07-30 18:14:53.712460494 +0000 UTC m=+4461.745270830"
Jul 30 18:14:54.695907 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:54.695868    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:14:54.697298 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:54.697248    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 30 18:14:55.699118 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:55.699077    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 30 18:14:57.521643 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:14:57.521598    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 30 18:15:00.704274 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:00.704234    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:15:00.704761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:00.704735    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 30 18:15:02.521308 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:02.521248    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 30 18:15:07.521438 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:07.521397    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 30 18:15:10.705916 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:10.705886    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:15:11.761812 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:11.761776    2569 generic.go:356] "Generic (PLEG): container finished" podID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerID="81f28272cb7b6b391cc47d9a7fc16bf014a3c9c986fbc7c87d112a2e2110f0e5" exitCode=137
Jul 30 18:15:11.762139 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:11.761850    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerDied","Data":"81f28272cb7b6b391cc47d9a7fc16bf014a3c9c986fbc7c87d112a2e2110f0e5"}
Jul 30 18:15:12.312294 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.312272    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:15:12.468269 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468167    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls\") pod \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") "
Jul 30 18:15:12.468269 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468220    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-cabundle-cert\") pod \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") "
Jul 30 18:15:12.468269 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468245    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ft68\" (UniqueName: \"kubernetes.io/projected/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-api-access-8ft68\") pod \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") "
Jul 30 18:15:12.468565 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kserve-provision-location\") pod \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") "
Jul 30 18:15:12.468565 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468308    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-rbac-proxy-sar-config\") pod \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\" (UID: \"3dae9f65-69d5-4400-bc2d-41d1e3f4b270\") "
Jul 30 18:15:12.468710 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468682    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3dae9f65-69d5-4400-bc2d-41d1e3f4b270" (UID: "3dae9f65-69d5-4400-bc2d-41d1e3f4b270"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:15:12.468752 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.468721    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3dae9f65-69d5-4400-bc2d-41d1e3f4b270" (UID: "3dae9f65-69d5-4400-bc2d-41d1e3f4b270"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:15:12.470297 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.470249    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dae9f65-69d5-4400-bc2d-41d1e3f4b270" (UID: "3dae9f65-69d5-4400-bc2d-41d1e3f4b270"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:15:12.470420 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.470359    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-api-access-8ft68" (OuterVolumeSpecName: "kube-api-access-8ft68") pod "3dae9f65-69d5-4400-bc2d-41d1e3f4b270" (UID: "3dae9f65-69d5-4400-bc2d-41d1e3f4b270"). InnerVolumeSpecName "kube-api-access-8ft68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:15:12.479268 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.479234    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dae9f65-69d5-4400-bc2d-41d1e3f4b270" (UID: "3dae9f65-69d5-4400-bc2d-41d1e3f4b270"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:15:12.568792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.568763    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:12.568792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.568789    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:12.568792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.568798    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:12.568979 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.568807    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:12.568979 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.568815    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ft68\" (UniqueName: \"kubernetes.io/projected/3dae9f65-69d5-4400-bc2d-41d1e3f4b270-kube-api-access-8ft68\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:12.767609 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.767522    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s" event={"ID":"3dae9f65-69d5-4400-bc2d-41d1e3f4b270","Type":"ContainerDied","Data":"d4c082bee03fafb1ec7efdfce01fd751da06d0ba02047fe10601e8b25950c6c9"}
Jul 30 18:15:12.767609 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.767555    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"
Jul 30 18:15:12.768067 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.767623    2569 scope.go:117] "RemoveContainer" containerID="09a637818f40e8d0079d4939d0128c2983439801ef25c44b98539226e02210bd"
Jul 30 18:15:12.775239 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.775219    2569 scope.go:117] "RemoveContainer" containerID="81f28272cb7b6b391cc47d9a7fc16bf014a3c9c986fbc7c87d112a2e2110f0e5"
Jul 30 18:15:12.782088 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.782069    2569 scope.go:117] "RemoveContainer" containerID="323f36801fff5a157ba53655e36bb76e5ccc715fdf7eb6371d33ea13ea1259d4"
Jul 30 18:15:12.794126 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.794102    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"]
Jul 30 18:15:12.797862 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:12.797834    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-sbh5s"]
Jul 30 18:15:14.652158 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:14.652120    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" path="/var/lib/kubelet/pods/3dae9f65-69d5-4400-bc2d-41d1e3f4b270/volumes"
Jul 30 18:15:27.974584 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:27.974500    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"]
Jul 30 18:15:27.974967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:27.974813    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" containerID="cri-o://5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18" gracePeriod=30
Jul 30 18:15:27.974967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:27.974801    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container" containerID="cri-o://a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e" gracePeriod=30
Jul 30 18:15:28.828897 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:28.828862    2569 generic.go:356] "Generic (PLEG): container finished" podID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerID="5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18" exitCode=2
Jul 30 18:15:28.829061 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:28.828941    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerDied","Data":"5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18"}
Jul 30 18:15:30.700028 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:30.699981    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 30 18:15:35.700093 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:35.700048    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 30 18:15:40.699766 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:40.699714    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 30 18:15:40.700327 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:40.699837    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:15:45.700365 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:45.700320    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 30 18:15:50.699925 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:50.699880    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 30 18:15:55.699779 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:55.699731    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 30 18:15:58.614130 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.614106    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:15:58.714343 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714302    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-cabundle-cert\") pod \"db9115da-ef1d-4f92-a84c-dbf9da656287\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") "
Jul 30 18:15:58.714542 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714352    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9115da-ef1d-4f92-a84c-dbf9da656287-proxy-tls\") pod \"db9115da-ef1d-4f92-a84c-dbf9da656287\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") "
Jul 30 18:15:58.714542 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714389    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9115da-ef1d-4f92-a84c-dbf9da656287-kserve-provision-location\") pod \"db9115da-ef1d-4f92-a84c-dbf9da656287\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") "
Jul 30 18:15:58.714542 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714452    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-rbac-proxy-sar-config\") pod \"db9115da-ef1d-4f92-a84c-dbf9da656287\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") "
Jul 30 18:15:58.714542 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714480    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qcm5\" (UniqueName: \"kubernetes.io/projected/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-api-access-5qcm5\") pod \"db9115da-ef1d-4f92-a84c-dbf9da656287\" (UID: \"db9115da-ef1d-4f92-a84c-dbf9da656287\") "
Jul 30 18:15:58.714841 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714805    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db9115da-ef1d-4f92-a84c-dbf9da656287" (UID: "db9115da-ef1d-4f92-a84c-dbf9da656287"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:15:58.714841 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.714838    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db9115da-ef1d-4f92-a84c-dbf9da656287" (UID: "db9115da-ef1d-4f92-a84c-dbf9da656287"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:15:58.716566 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.716532    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-api-access-5qcm5" (OuterVolumeSpecName: "kube-api-access-5qcm5") pod "db9115da-ef1d-4f92-a84c-dbf9da656287" (UID: "db9115da-ef1d-4f92-a84c-dbf9da656287"). InnerVolumeSpecName "kube-api-access-5qcm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:15:58.716653 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.716611    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9115da-ef1d-4f92-a84c-dbf9da656287-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db9115da-ef1d-4f92-a84c-dbf9da656287" (UID: "db9115da-ef1d-4f92-a84c-dbf9da656287"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:15:58.724360 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.724333    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9115da-ef1d-4f92-a84c-dbf9da656287-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db9115da-ef1d-4f92-a84c-dbf9da656287" (UID: "db9115da-ef1d-4f92-a84c-dbf9da656287"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:15:58.815318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.815210    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:58.815318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.815239    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9115da-ef1d-4f92-a84c-dbf9da656287-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:58.815318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.815270    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9115da-ef1d-4f92-a84c-dbf9da656287-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:58.815318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.815283    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:58.815318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.815296    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qcm5\" (UniqueName: \"kubernetes.io/projected/db9115da-ef1d-4f92-a84c-dbf9da656287-kube-api-access-5qcm5\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:15:58.935406 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.935371    2569 generic.go:356] "Generic (PLEG): container finished" podID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerID="a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e" exitCode=137
Jul 30 18:15:58.935583 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.935440    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"
Jul 30 18:15:58.935583 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.935454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerDied","Data":"a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e"}
Jul 30 18:15:58.935583 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.935519    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2" event={"ID":"db9115da-ef1d-4f92-a84c-dbf9da656287","Type":"ContainerDied","Data":"a236aca8f55a783f321a5bfddfdcbcddba68814e5c3272ae9f5eeaec8f597693"}
Jul 30 18:15:58.935583 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.935550    2569 scope.go:117] "RemoveContainer" containerID="5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18"
Jul 30 18:15:58.944035 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.944017    2569 scope.go:117] "RemoveContainer" containerID="a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e"
Jul 30 18:15:58.950945 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.950925    2569 scope.go:117] "RemoveContainer" containerID="231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2"
Jul 30 18:15:58.957637 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.957616    2569 scope.go:117] "RemoveContainer" containerID="5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18"
Jul 30 18:15:58.958058 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:15:58.958015    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18\": container with ID starting with 5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18 not found: ID does not exist" containerID="5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18"
Jul 30 18:15:58.958215 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.958057    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18"} err="failed to get container status \"5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18\": rpc error: code = NotFound desc = could not find container \"5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18\": container with ID starting with 5a66d889878f316059e6f6ff13b857c1d0ba5e90e7db04ef46dd41da0631bf18 not found: ID does not exist"
Jul 30 18:15:58.958215 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.958084    2569 scope.go:117] "RemoveContainer" containerID="a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e"
Jul 30 18:15:58.958462 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:15:58.958405    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e\": container with ID starting with a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e not found: ID does not exist" containerID="a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e"
Jul 30 18:15:58.958462 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.958435    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e"} err="failed to get container status \"a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e\": rpc error: code = NotFound desc = could not find container \"a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e\": container with ID starting with a3d896d50f921a6beaf770aa46cb05ca311276adb16aa01661507366b827169e not found: ID does not exist"
Jul 30 18:15:58.958462 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.958459    2569 scope.go:117] "RemoveContainer" containerID="231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2"
Jul 30 18:15:58.958789 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:15:58.958767    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2\": container with ID starting with 231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2 not found: ID does not exist" containerID="231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2"
Jul 30 18:15:58.958864 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.958802    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2"} err="failed to get container status \"231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2\": rpc error: code = NotFound desc = could not find container \"231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2\": container with ID starting with 231bc56bfdfee7d324b7046e08c54f3fe426538664cb960941101af3fb4c38b2 not found: ID does not exist"
Jul 30 18:15:58.960102 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.960082    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"]
Jul 30 18:15:58.964536 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:15:58.964515    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-k67g2"]
Jul 30 18:16:00.652851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:16:00.652807    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" path="/var/lib/kubelet/pods/db9115da-ef1d-4f92-a84c-dbf9da656287/volumes"
Jul 30 18:17:49.837709 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837662    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"]
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837947    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837959    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837973    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837979    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837985    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837991    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.837996    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838001    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838008    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="storage-initializer"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838014    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="storage-initializer"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838020    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="storage-initializer"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838025    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="storage-initializer"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838085    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kserve-container"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838091    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kserve-container"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838097    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9115da-ef1d-4f92-a84c-dbf9da656287" containerName="kube-rbac-proxy"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.838103    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dae9f65-69d5-4400-bc2d-41d1e3f4b270" containerName="kube-rbac-proxy"
Jul 30 18:17:49.844644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.844530    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:49.847302 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.847280    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:17:49.847429 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.847316    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 30 18:17:49.847631 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.847615    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:17:49.847756 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.847712    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:17:49.848362 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.848341    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 30 18:17:49.848502 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.848402    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:17:49.848502 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.848405    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:17:49.851604 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.851583    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"]
Jul 30 18:17:49.974852 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.974810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1831094c-f80a-481f-a43e-b8ac1d2de686-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:49.974852 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.974852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1831094c-f80a-481f-a43e-b8ac1d2de686-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:49.975096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.974876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:49.975096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.975015    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5tr\" (UniqueName: \"kubernetes.io/projected/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-api-access-4l5tr\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:49.975096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:49.975057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5tr\" (UniqueName: \"kubernetes.io/projected/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-api-access-4l5tr\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076300 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076300 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1831094c-f80a-481f-a43e-b8ac1d2de686-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076300 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1831094c-f80a-481f-a43e-b8ac1d2de686-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076300 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076226    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076649 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076627    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1831094c-f80a-481f-a43e-b8ac1d2de686-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076849 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.076889 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.076863    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.078624 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.078599    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1831094c-f80a-481f-a43e-b8ac1d2de686-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.084872 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.084849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5tr\" (UniqueName: \"kubernetes.io/projected/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-api-access-4l5tr\") pod \"isvc-xgboost-predictor-7566d65bf7-nqflj\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.155905 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.155817    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:17:50.281795 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.281768    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"]
Jul 30 18:17:50.284374 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:17:50.284348    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1831094c_f80a_481f_a43e_b8ac1d2de686.slice/crio-88b1cd88aeb980eba217253f5fdc8ee8d59b49de20f78bf9ad974b0fd68bebd3 WatchSource:0}: Error finding container 88b1cd88aeb980eba217253f5fdc8ee8d59b49de20f78bf9ad974b0fd68bebd3: Status 404 returned error can't find the container with id 88b1cd88aeb980eba217253f5fdc8ee8d59b49de20f78bf9ad974b0fd68bebd3
Jul 30 18:17:50.336520 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:50.336492    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerStarted","Data":"88b1cd88aeb980eba217253f5fdc8ee8d59b49de20f78bf9ad974b0fd68bebd3"}
Jul 30 18:17:51.341952 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:51.341916    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerStarted","Data":"177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4"}
Jul 30 18:17:54.354979 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:54.354941    2569 generic.go:356] "Generic (PLEG): container finished" podID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerID="177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4" exitCode=0
Jul 30 18:17:54.355451 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:17:54.355018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerDied","Data":"177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4"}
Jul 30 18:18:15.442618 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:15.442585    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerStarted","Data":"b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3"}
Jul 30 18:18:15.442996 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:15.442648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerStarted","Data":"8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b"}
Jul 30 18:18:15.442996 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:15.442838    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:18:15.464830 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:15.464782    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podStartSLOduration=1.735247002 podStartE2EDuration="26.464767596s" totalImagesPullingTime="20.825740664s" totalInitContainerRuntime="3.90377993s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:18:15.463369421 +0000 UTC m=+4663.496179760" watchObservedRunningTime="2026-07-30 18:18:15.464767596 +0000 UTC m=+4663.497577933"
Jul 30 18:18:16.446761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:16.446731    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:18:16.448045 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:16.448019    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:18:17.450552 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:17.450518    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:18:22.455282 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:22.455240    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:18:22.455776 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:22.455750    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:18:32.456618 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:32.456537    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:18:42.455864 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:42.455818    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:18:52.455725 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:18:52.455684    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:19:02.456383 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:02.456337    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:19:12.456226 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:12.456186    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:19:22.457163 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:22.457133    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:19:29.964058 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:29.964026    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"]
Jul 30 18:19:29.964482 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:29.964267    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" containerID="cri-o://8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b" gracePeriod=30
Jul 30 18:19:29.964482 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:29.964318    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kube-rbac-proxy" containerID="cri-o://b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3" gracePeriod=30
Jul 30 18:19:30.726669 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:30.726633    2569 generic.go:356] "Generic (PLEG): container finished" podID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerID="b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3" exitCode=2
Jul 30 18:19:30.726855 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:30.726713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerDied","Data":"b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3"}
Jul 30 18:19:32.451142 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:32.451103    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 30 18:19:32.456685 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:32.456656    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 30 18:19:33.498047 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.498024    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:19:33.573088 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573052    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1831094c-f80a-481f-a43e-b8ac1d2de686-proxy-tls\") pod \"1831094c-f80a-481f-a43e-b8ac1d2de686\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") "
Jul 30 18:19:33.573287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573103    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1831094c-f80a-481f-a43e-b8ac1d2de686-kserve-provision-location\") pod \"1831094c-f80a-481f-a43e-b8ac1d2de686\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") "
Jul 30 18:19:33.573287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573128    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-cabundle-cert\") pod \"1831094c-f80a-481f-a43e-b8ac1d2de686\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") "
Jul 30 18:19:33.573287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573183    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-rbac-proxy-sar-config\") pod \"1831094c-f80a-481f-a43e-b8ac1d2de686\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") "
Jul 30 18:19:33.573287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573207    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l5tr\" (UniqueName: \"kubernetes.io/projected/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-api-access-4l5tr\") pod \"1831094c-f80a-481f-a43e-b8ac1d2de686\" (UID: \"1831094c-f80a-481f-a43e-b8ac1d2de686\") "
Jul 30 18:19:33.573560 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573506    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1831094c-f80a-481f-a43e-b8ac1d2de686-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1831094c-f80a-481f-a43e-b8ac1d2de686" (UID: "1831094c-f80a-481f-a43e-b8ac1d2de686"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:19:33.573560 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573521    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1831094c-f80a-481f-a43e-b8ac1d2de686" (UID: "1831094c-f80a-481f-a43e-b8ac1d2de686"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:19:33.573652 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.573581    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1831094c-f80a-481f-a43e-b8ac1d2de686" (UID: "1831094c-f80a-481f-a43e-b8ac1d2de686"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:19:33.575031 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.575006    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1831094c-f80a-481f-a43e-b8ac1d2de686-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1831094c-f80a-481f-a43e-b8ac1d2de686" (UID: "1831094c-f80a-481f-a43e-b8ac1d2de686"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:19:33.575191 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.575171    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-api-access-4l5tr" (OuterVolumeSpecName: "kube-api-access-4l5tr") pod "1831094c-f80a-481f-a43e-b8ac1d2de686" (UID: "1831094c-f80a-481f-a43e-b8ac1d2de686"). InnerVolumeSpecName "kube-api-access-4l5tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:19:33.674276 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.674231    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1831094c-f80a-481f-a43e-b8ac1d2de686-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:19:33.674276 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.674272    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1831094c-f80a-481f-a43e-b8ac1d2de686-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:19:33.674276 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.674282    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:19:33.674514 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.674291    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:19:33.674514 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.674300    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l5tr\" (UniqueName: \"kubernetes.io/projected/1831094c-f80a-481f-a43e-b8ac1d2de686-kube-api-access-4l5tr\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:19:33.739608 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.739575    2569 generic.go:356] "Generic (PLEG): container finished" podID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerID="8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b" exitCode=0
Jul 30 18:19:33.739761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.739629    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"
Jul 30 18:19:33.739761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.739650    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerDied","Data":"8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b"}
Jul 30 18:19:33.739761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.739700    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj" event={"ID":"1831094c-f80a-481f-a43e-b8ac1d2de686","Type":"ContainerDied","Data":"88b1cd88aeb980eba217253f5fdc8ee8d59b49de20f78bf9ad974b0fd68bebd3"}
Jul 30 18:19:33.739761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.739727    2569 scope.go:117] "RemoveContainer" containerID="b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3"
Jul 30 18:19:33.748218 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.748202    2569 scope.go:117] "RemoveContainer" containerID="8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b"
Jul 30 18:19:33.755072 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.755050    2569 scope.go:117] "RemoveContainer" containerID="177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4"
Jul 30 18:19:33.762335 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.762316    2569 scope.go:117] "RemoveContainer" containerID="b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3"
Jul 30 18:19:33.762590 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.762569    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"]
Jul 30 18:19:33.762652 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:19:33.762573    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3\": container with ID starting with b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3 not found: ID does not exist" containerID="b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3"
Jul 30 18:19:33.762652 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.762627    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3"} err="failed to get container status \"b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3\": rpc error: code = NotFound desc = could not find container \"b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3\": container with ID starting with b400ccf73ad1d49e3295bd390fb9a5541065f077d62a580690c8e11fa84c9ab3 not found: ID does not exist"
Jul 30 18:19:33.762652 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.762646    2569 scope.go:117] "RemoveContainer" containerID="8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b"
Jul 30 18:19:33.762908 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:19:33.762889    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b\": container with ID starting with 8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b not found: ID does not exist" containerID="8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b"
Jul 30 18:19:33.762961 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.762913    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b"} err="failed to get container status \"8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b\": rpc error: code = NotFound desc = could not find container \"8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b\": container with ID starting with 8f43e895a625990a477823201e0fbb68448cd615346362c217b7b231bf9bfe5b not found: ID does not exist"
Jul 30 18:19:33.762961 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.762929    2569 scope.go:117] "RemoveContainer" containerID="177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4"
Jul 30 18:19:33.763172 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:19:33.763156    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4\": container with ID starting with 177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4 not found: ID does not exist" containerID="177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4"
Jul 30 18:19:33.763224 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.763183    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4"} err="failed to get container status \"177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4\": rpc error: code = NotFound desc = could not find container \"177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4\": container with ID starting with 177eb297d641e2d83ddea6e38d31803d6f8f629199bd7e7f992be3776910b8f4 not found: ID does not exist"
Jul 30 18:19:33.768217 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:33.768195    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-nqflj"]
Jul 30 18:19:34.652270 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:19:34.652215    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" path="/var/lib/kubelet/pods/1831094c-f80a-481f-a43e-b8ac1d2de686/volumes"
Jul 30 18:21:25.647277 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647179    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"]
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647464    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kube-rbac-proxy"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647476    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kube-rbac-proxy"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647486    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="storage-initializer"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647491    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="storage-initializer"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647501    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647508    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647558    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kserve-container"
Jul 30 18:21:25.647728 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.647565    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1831094c-f80a-481f-a43e-b8ac1d2de686" containerName="kube-rbac-proxy"
Jul 30 18:21:25.649801 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.649785    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.652491 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.652463    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:21:25.652623 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.652501    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:21:25.652623 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.652507    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:21:25.653415 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.653396    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:21:25.653532 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.653432    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 30 18:21:25.653532 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.653445    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:21:25.653532 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.653433    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 30 18:21:25.661998 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.661978    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"]
Jul 30 18:21:25.743107 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.743074    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cba462-c451-45a1-9871-f719a9832a2f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.743314 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.743117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.743314 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.743137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.743314 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.743228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhl7\" (UniqueName: \"kubernetes.io/projected/a8cba462-c451-45a1-9871-f719a9832a2f-kube-api-access-dqhl7\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.743314 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.743293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.843602 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.843565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhl7\" (UniqueName: \"kubernetes.io/projected/a8cba462-c451-45a1-9871-f719a9832a2f-kube-api-access-dqhl7\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.843774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.843610    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.843774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.843644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cba462-c451-45a1-9871-f719a9832a2f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.843774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.843673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.843774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.843691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.843941 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:21:25.843819    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 30 18:21:25.843941 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:21:25.843877    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls podName:a8cba462-c451-45a1-9871-f719a9832a2f nodeName:}" failed. No retries permitted until 2026-07-30 18:21:26.343858479 +0000 UTC m=+4854.376668793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-k87sw" (UID: "a8cba462-c451-45a1-9871-f719a9832a2f") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 30 18:21:25.844114 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.844089    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cba462-c451-45a1-9871-f719a9832a2f-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.844436 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.844413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.844481 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.844434    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:25.855066 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:25.855047    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhl7\" (UniqueName: \"kubernetes.io/projected/a8cba462-c451-45a1-9871-f719a9832a2f-kube-api-access-dqhl7\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:26.347384 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:26.347345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:26.349703 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:26.349666    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-k87sw\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:26.559735 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:26.559697    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:26.685423 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:26.685396    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"]
Jul 30 18:21:26.687292 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:21:26.687248    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cba462_c451_45a1_9871_f719a9832a2f.slice/crio-4e064286a152a62d3163a2822d8cbbd732273bd428aecddfa728f1de2515835f WatchSource:0}: Error finding container 4e064286a152a62d3163a2822d8cbbd732273bd428aecddfa728f1de2515835f: Status 404 returned error can't find the container with id 4e064286a152a62d3163a2822d8cbbd732273bd428aecddfa728f1de2515835f
Jul 30 18:21:27.146900 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:27.146861    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerStarted","Data":"b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b"}
Jul 30 18:21:27.147055 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:27.146929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerStarted","Data":"4e064286a152a62d3163a2822d8cbbd732273bd428aecddfa728f1de2515835f"}
Jul 30 18:21:31.163856 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:31.163822    2569 generic.go:356] "Generic (PLEG): container finished" podID="a8cba462-c451-45a1-9871-f719a9832a2f" containerID="b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b" exitCode=0
Jul 30 18:21:31.164244 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:31.163890    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerDied","Data":"b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b"}
Jul 30 18:21:32.169464 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:32.169432    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerStarted","Data":"fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef"}
Jul 30 18:21:32.169877 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:32.169493    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerStarted","Data":"0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c"}
Jul 30 18:21:32.169877 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:32.169673    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:32.188978 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:32.188932    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podStartSLOduration=3.574255955 podStartE2EDuration="7.188916044s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.614660089s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:21:32.18798842 +0000 UTC m=+4860.220798761" watchObservedRunningTime="2026-07-30 18:21:32.188916044 +0000 UTC m=+4860.221726380"
Jul 30 18:21:33.173374 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:33.173343    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:33.174395 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:33.174372    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:21:34.177036 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:34.176994    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:21:39.181189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:39.181152    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:21:39.181821 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:39.181794    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:21:49.181788 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:49.181748    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:21:59.181866 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:21:59.181823    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:22:09.182168 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:09.182129    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:22:19.182010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:19.181971    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:22:29.182679 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:29.182629    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:22:39.182422 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:39.182382    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:22:45.740434 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:45.740388    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"]
Jul 30 18:22:45.740910 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:45.740693    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" containerID="cri-o://0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c" gracePeriod=30
Jul 30 18:22:45.740910 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:45.740766    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kube-rbac-proxy" containerID="cri-o://fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef" gracePeriod=30
Jul 30 18:22:46.443818 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:46.443785    2569 generic.go:356] "Generic (PLEG): container finished" podID="a8cba462-c451-45a1-9871-f719a9832a2f" containerID="fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef" exitCode=2
Jul 30 18:22:46.443991 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:46.443853    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerDied","Data":"fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef"}
Jul 30 18:22:49.177527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.177486    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 30 18:22:49.181737 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.181704    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 30 18:22:49.288098 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.288070    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:22:49.422795 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.422698    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls\") pod \"a8cba462-c451-45a1-9871-f719a9832a2f\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") "
Jul 30 18:22:49.422795 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.422759    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqhl7\" (UniqueName: \"kubernetes.io/projected/a8cba462-c451-45a1-9871-f719a9832a2f-kube-api-access-dqhl7\") pod \"a8cba462-c451-45a1-9871-f719a9832a2f\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") "
Jul 30 18:22:49.423028 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.422802    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-cabundle-cert\") pod \"a8cba462-c451-45a1-9871-f719a9832a2f\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") "
Jul 30 18:22:49.423028 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.422829    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-kube-rbac-proxy-sar-config\") pod \"a8cba462-c451-45a1-9871-f719a9832a2f\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") "
Jul 30 18:22:49.423028 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.422847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cba462-c451-45a1-9871-f719a9832a2f-kserve-provision-location\") pod \"a8cba462-c451-45a1-9871-f719a9832a2f\" (UID: \"a8cba462-c451-45a1-9871-f719a9832a2f\") "
Jul 30 18:22:49.423207 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.423184    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8cba462-c451-45a1-9871-f719a9832a2f" (UID: "a8cba462-c451-45a1-9871-f719a9832a2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:22:49.423294 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.423202    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8cba462-c451-45a1-9871-f719a9832a2f" (UID: "a8cba462-c451-45a1-9871-f719a9832a2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:22:49.423294 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.423216    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cba462-c451-45a1-9871-f719a9832a2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8cba462-c451-45a1-9871-f719a9832a2f" (UID: "a8cba462-c451-45a1-9871-f719a9832a2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:22:49.424809 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.424785    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8cba462-c451-45a1-9871-f719a9832a2f" (UID: "a8cba462-c451-45a1-9871-f719a9832a2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:22:49.424872 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.424826    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cba462-c451-45a1-9871-f719a9832a2f-kube-api-access-dqhl7" (OuterVolumeSpecName: "kube-api-access-dqhl7") pod "a8cba462-c451-45a1-9871-f719a9832a2f" (UID: "a8cba462-c451-45a1-9871-f719a9832a2f"). InnerVolumeSpecName "kube-api-access-dqhl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:22:49.455772 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.455744    2569 generic.go:356] "Generic (PLEG): container finished" podID="a8cba462-c451-45a1-9871-f719a9832a2f" containerID="0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c" exitCode=0
Jul 30 18:22:49.455920 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.455797    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"
Jul 30 18:22:49.455920 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.455824    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerDied","Data":"0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c"}
Jul 30 18:22:49.455920 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.455886    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw" event={"ID":"a8cba462-c451-45a1-9871-f719a9832a2f","Type":"ContainerDied","Data":"4e064286a152a62d3163a2822d8cbbd732273bd428aecddfa728f1de2515835f"}
Jul 30 18:22:49.455920 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.455915    2569 scope.go:117] "RemoveContainer" containerID="fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef"
Jul 30 18:22:49.464766 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.464749    2569 scope.go:117] "RemoveContainer" containerID="0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c"
Jul 30 18:22:49.471719 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.471703    2569 scope.go:117] "RemoveContainer" containerID="b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b"
Jul 30 18:22:49.479141 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.479120    2569 scope.go:117] "RemoveContainer" containerID="fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef"
Jul 30 18:22:49.479439 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:22:49.479411    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef\": container with ID starting with fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef not found: ID does not exist" containerID="fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef"
Jul 30 18:22:49.479506 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.479451    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef"} err="failed to get container status \"fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef\": rpc error: code = NotFound desc = could not find container \"fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef\": container with ID starting with fa991ccfae3a26490b50944819516a5dc844de17127bc1f9a1481d867fca64ef not found: ID does not exist"
Jul 30 18:22:49.479506 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.479474    2569 scope.go:117] "RemoveContainer" containerID="0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c"
Jul 30 18:22:49.479718 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:22:49.479693    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c\": container with ID starting with 0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c not found: ID does not exist" containerID="0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c"
Jul 30 18:22:49.479837 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.479721    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c"} err="failed to get container status \"0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c\": rpc error: code = NotFound desc = could not find container \"0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c\": container with ID starting with 0c1442d61d3dffd88c0f77072e860eaea45d09809621c0f3556abc4008afdd0c not found: ID does not exist"
Jul 30 18:22:49.479837 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.479738    2569 scope.go:117] "RemoveContainer" containerID="b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b"
Jul 30 18:22:49.479956 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:22:49.479940    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b\": container with ID starting with b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b not found: ID does not exist" containerID="b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b"
Jul 30 18:22:49.479997 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.479959    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b"} err="failed to get container status \"b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b\": rpc error: code = NotFound desc = could not find container \"b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b\": container with ID starting with b934ceaefac9a3f66d141382eda05799cb25624bbca5d7b29cf8651804c3223b not found: ID does not exist"
Jul 30 18:22:49.480154 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.480137    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"]
Jul 30 18:22:49.484570 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.484543    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-k87sw"]
Jul 30 18:22:49.523670 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.523632    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhl7\" (UniqueName: \"kubernetes.io/projected/a8cba462-c451-45a1-9871-f719a9832a2f-kube-api-access-dqhl7\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:22:49.523670 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.523664    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:22:49.523670 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.523675    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8cba462-c451-45a1-9871-f719a9832a2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:22:49.523898 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.523684    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8cba462-c451-45a1-9871-f719a9832a2f-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:22:49.523898 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:49.523693    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8cba462-c451-45a1-9871-f719a9832a2f-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:22:50.651586 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:22:50.651556    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" path="/var/lib/kubelet/pods/a8cba462-c451-45a1-9871-f719a9832a2f/volumes"
Jul 30 18:23:56.201352 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201318    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"]
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201596    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="storage-initializer"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201607    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="storage-initializer"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201616    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201621    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201640    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kube-rbac-proxy"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201645    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kube-rbac-proxy"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201682    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kserve-container"
Jul 30 18:23:56.201814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.201698    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cba462-c451-45a1-9871-f719a9832a2f" containerName="kube-rbac-proxy"
Jul 30 18:23:56.204736 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.204717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.207537 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.207515    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:23:56.208686 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.208671    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:23:56.208768 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.208695    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 30 18:23:56.208768 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.208721    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:23:56.208890 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.208678    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 30 18:23:56.208890 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.208784    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:23:56.208890 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.208834    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 30 18:23:56.216030 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.216009    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"]
Jul 30 18:23:56.248410 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.248381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.248527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.248421    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.248527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.248483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khs58\" (UniqueName: \"kubernetes.io/projected/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-api-access-khs58\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.248617 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.248532    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebecf15-6035-46a5-9203-bd66a1ed50a7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.248617 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.248563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.349597 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.349560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.349764 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.349612    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.349764 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.349649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.349764 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.349682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khs58\" (UniqueName: \"kubernetes.io/projected/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-api-access-khs58\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.349923 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.349833    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebecf15-6035-46a5-9203-bd66a1ed50a7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.350166 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.350143    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.350342 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.350324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.350411 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.350360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.352214 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.352196    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebecf15-6035-46a5-9203-bd66a1ed50a7-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.359759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.359734    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khs58\" (UniqueName: \"kubernetes.io/projected/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-api-access-khs58\") pod \"isvc-xgboost-v2-predictor-fcc668bc-k7pm4\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.514285 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.514162    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:23:56.642573 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.642414    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"]
Jul 30 18:23:56.645098 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:23:56.645072    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebecf15_6035_46a5_9203_bd66a1ed50a7.slice/crio-c0a4607dedbe3d9076fb383b42d1229694df78547be93ac606e237e7bd39f160 WatchSource:0}: Error finding container c0a4607dedbe3d9076fb383b42d1229694df78547be93ac606e237e7bd39f160: Status 404 returned error can't find the container with id c0a4607dedbe3d9076fb383b42d1229694df78547be93ac606e237e7bd39f160
Jul 30 18:23:56.694694 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:56.694667    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerStarted","Data":"c0a4607dedbe3d9076fb383b42d1229694df78547be93ac606e237e7bd39f160"}
Jul 30 18:23:57.700183 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:23:57.700141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerStarted","Data":"cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137"}
Jul 30 18:24:00.712309 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:00.712196    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerID="cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137" exitCode=0
Jul 30 18:24:00.712309 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:00.712279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerDied","Data":"cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137"}
Jul 30 18:24:01.718236 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:01.718199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerStarted","Data":"ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b"}
Jul 30 18:24:01.718666 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:01.718309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerStarted","Data":"6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926"}
Jul 30 18:24:01.718666 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:01.718598    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:24:01.718791 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:01.718761    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:24:01.720164 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:01.720137    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:01.741085 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:01.741041    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podStartSLOduration=1.978929329 podStartE2EDuration="5.741027071s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.762097742s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:24:01.739689899 +0000 UTC m=+5009.772500250" watchObservedRunningTime="2026-07-30 18:24:01.741027071 +0000 UTC m=+5009.773837409"
Jul 30 18:24:02.721701 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:02.721657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:03.725282 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:03.725213    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:08.730119 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:08.730090    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:24:08.730618 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:08.730589    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:18.730992 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:18.730950    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:28.730841 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:28.730758    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:38.730695 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:38.730658    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:48.731095 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:48.731052    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:24:58.730826 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:24:58.730788    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:25:08.731976 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:08.731944    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:25:16.291356 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:16.291317    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"]
Jul 30 18:25:16.291798 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:16.291555    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" containerID="cri-o://6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926" gracePeriod=30
Jul 30 18:25:16.291798 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:16.291581    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kube-rbac-proxy" containerID="cri-o://ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b" gracePeriod=30
Jul 30 18:25:16.997563 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:16.997530    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerID="ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b" exitCode=2
Jul 30 18:25:16.997752 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:16.997580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerDied","Data":"ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b"}
Jul 30 18:25:18.725895 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:18.725852    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 30 18:25:18.731308 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:18.731246    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 30 18:25:19.927480 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:19.927456    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:25:20.009915 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.009814    2569 generic.go:356] "Generic (PLEG): container finished" podID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerID="6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926" exitCode=0
Jul 30 18:25:20.009915 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.009871    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerDied","Data":"6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926"}
Jul 30 18:25:20.009915 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.009880    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"
Jul 30 18:25:20.010189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.009931    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4" event={"ID":"1ebecf15-6035-46a5-9203-bd66a1ed50a7","Type":"ContainerDied","Data":"c0a4607dedbe3d9076fb383b42d1229694df78547be93ac606e237e7bd39f160"}
Jul 30 18:25:20.010189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.009968    2569 scope.go:117] "RemoveContainer" containerID="ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b"
Jul 30 18:25:20.017789 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.017772    2569 scope.go:117] "RemoveContainer" containerID="6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926"
Jul 30 18:25:20.024413 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.024394    2569 scope.go:117] "RemoveContainer" containerID="cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137"
Jul 30 18:25:20.027439 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027420    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-cabundle-cert\") pod \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") "
Jul 30 18:25:20.027531 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027453    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khs58\" (UniqueName: \"kubernetes.io/projected/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-api-access-khs58\") pod \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") "
Jul 30 18:25:20.027531 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027475    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebecf15-6035-46a5-9203-bd66a1ed50a7-proxy-tls\") pod \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") "
Jul 30 18:25:20.027642 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027541    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kserve-provision-location\") pod \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") "
Jul 30 18:25:20.027642 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027570    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-rbac-proxy-sar-config\") pod \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\" (UID: \"1ebecf15-6035-46a5-9203-bd66a1ed50a7\") "
Jul 30 18:25:20.027907 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027872    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ebecf15-6035-46a5-9203-bd66a1ed50a7" (UID: "1ebecf15-6035-46a5-9203-bd66a1ed50a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:25:20.027907 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027895    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ebecf15-6035-46a5-9203-bd66a1ed50a7" (UID: "1ebecf15-6035-46a5-9203-bd66a1ed50a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:25:20.028042 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.027963    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ebecf15-6035-46a5-9203-bd66a1ed50a7" (UID: "1ebecf15-6035-46a5-9203-bd66a1ed50a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:25:20.029523 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.029498    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebecf15-6035-46a5-9203-bd66a1ed50a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ebecf15-6035-46a5-9203-bd66a1ed50a7" (UID: "1ebecf15-6035-46a5-9203-bd66a1ed50a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:25:20.029627 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.029603    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-api-access-khs58" (OuterVolumeSpecName: "kube-api-access-khs58") pod "1ebecf15-6035-46a5-9203-bd66a1ed50a7" (UID: "1ebecf15-6035-46a5-9203-bd66a1ed50a7"). InnerVolumeSpecName "kube-api-access-khs58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:25:20.031706 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.031684    2569 scope.go:117] "RemoveContainer" containerID="ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b"
Jul 30 18:25:20.031958 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:25:20.031939    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b\": container with ID starting with ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b not found: ID does not exist" containerID="ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b"
Jul 30 18:25:20.032033 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.031964    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b"} err="failed to get container status \"ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b\": rpc error: code = NotFound desc = could not find container \"ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b\": container with ID starting with ded51d4ec17103e37b8ee0798aa91af4282758f6bcf9b537c58423559cfeaa1b not found: ID does not exist"
Jul 30 18:25:20.032033 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.031981    2569 scope.go:117] "RemoveContainer" containerID="6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926"
Jul 30 18:25:20.032212 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:25:20.032191    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926\": container with ID starting with 6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926 not found: ID does not exist" containerID="6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926"
Jul 30 18:25:20.032291 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.032218    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926"} err="failed to get container status \"6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926\": rpc error: code = NotFound desc = could not find container \"6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926\": container with ID starting with 6c77663fca2c6ca9c7fac345284fe7d486059f5b0b92d7ae1462d13e01bae926 not found: ID does not exist"
Jul 30 18:25:20.032291 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.032236    2569 scope.go:117] "RemoveContainer" containerID="cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137"
Jul 30 18:25:20.032515 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:25:20.032497    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137\": container with ID starting with cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137 not found: ID does not exist" containerID="cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137"
Jul 30 18:25:20.032575 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.032518    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137"} err="failed to get container status \"cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137\": rpc error: code = NotFound desc = could not find container \"cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137\": container with ID starting with cabe1ed95064f73afad4e84346681f30ef1b50a91574d201c1199420f214d137 not found: ID does not exist"
Jul 30 18:25:20.129064 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.129031    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:25:20.129064 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.129059    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khs58\" (UniqueName: \"kubernetes.io/projected/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-api-access-khs58\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:25:20.129064 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.129069    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebecf15-6035-46a5-9203-bd66a1ed50a7-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:25:20.129288 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.129079    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:25:20.129288 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.129088    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebecf15-6035-46a5-9203-bd66a1ed50a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:25:20.337329 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.337294    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"]
Jul 30 18:25:20.341561 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.341535    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-k7pm4"]
Jul 30 18:25:20.651290 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:20.651190    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" path="/var/lib/kubelet/pods/1ebecf15-6035-46a5-9203-bd66a1ed50a7/volumes"
Jul 30 18:25:21.443493 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443460    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"]
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443824    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="storage-initializer"
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443844    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="storage-initializer"
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443869    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container"
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443878    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container"
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443895    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kube-rbac-proxy"
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443903    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kube-rbac-proxy"
Jul 30 18:25:21.443967 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443961    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kube-rbac-proxy"
Jul 30 18:25:21.444370 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.443974    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebecf15-6035-46a5-9203-bd66a1ed50a7" containerName="kserve-container"
Jul 30 18:25:21.448724 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.448703    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.451209 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.451180    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:25:21.451340 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.451211    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:25:21.451340 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.451181    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 30 18:25:21.451340 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.451236    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 30 18:25:21.451340 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.451240    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 30 18:25:21.451499 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.451369    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:25:21.457474 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.457453    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"]
Jul 30 18:25:21.538862 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.538820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.539048 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.538880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.539048 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.538967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.539048 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.539006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wrk\" (UniqueName: \"kubernetes.io/projected/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-api-access-c9wrk\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.639815 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.639774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.639985 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.639826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.639985 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:25:21.639936    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 30 18:25:21.640083 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.639999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.640083 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:25:21.640011    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls podName:51fc2544-11e0-4b76-947b-6043a1bc8d7e nodeName:}" failed. No retries permitted until 2026-07-30 18:25:22.139995281 +0000 UTC m=+5090.172805596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls") pod "isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" (UID: "51fc2544-11e0-4b76-947b-6043a1bc8d7e") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 30 18:25:21.640083 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.640042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wrk\" (UniqueName: \"kubernetes.io/projected/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-api-access-c9wrk\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.640337 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.640164    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.640699 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.640675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:21.651030 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:21.651007    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wrk\" (UniqueName: \"kubernetes.io/projected/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-api-access-c9wrk\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:22.143030 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:22.142985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:22.145417 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:22.145391    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:22.359797 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:22.359757    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:22.491739 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:22.491704    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"]
Jul 30 18:25:22.495131 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:25:22.495105    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fc2544_11e0_4b76_947b_6043a1bc8d7e.slice/crio-ed57dfafbe04b903fa8a5ad8e7024c79245cd6a9cf7af71c6b5432b625138bef WatchSource:0}: Error finding container ed57dfafbe04b903fa8a5ad8e7024c79245cd6a9cf7af71c6b5432b625138bef: Status 404 returned error can't find the container with id ed57dfafbe04b903fa8a5ad8e7024c79245cd6a9cf7af71c6b5432b625138bef
Jul 30 18:25:23.023105 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:23.023074    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerStarted","Data":"59cfb7e66fb9b51eb3e5ff951070c666501ce4bfd38491509e4e2f98a27397a0"}
Jul 30 18:25:23.023302 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:23.023136    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerStarted","Data":"ed57dfafbe04b903fa8a5ad8e7024c79245cd6a9cf7af71c6b5432b625138bef"}
Jul 30 18:25:24.028510 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:24.028470    2569 generic.go:356] "Generic (PLEG): container finished" podID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerID="59cfb7e66fb9b51eb3e5ff951070c666501ce4bfd38491509e4e2f98a27397a0" exitCode=0
Jul 30 18:25:24.028899 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:24.028550    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerDied","Data":"59cfb7e66fb9b51eb3e5ff951070c666501ce4bfd38491509e4e2f98a27397a0"}
Jul 30 18:25:25.034391 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:25.034352    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerStarted","Data":"0adbb5c4fde5534e97bba8b6ba5ab424d3816f808fed8aca14c6155a720c4204"}
Jul 30 18:25:25.034810 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:25.034447    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerStarted","Data":"2d393010e9cb1edad1a1a3343350e436be59df57ecd16cd7cb9f5e8d8c304d58"}
Jul 30 18:25:25.034810 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:25.034618    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:25.034810 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:25.034666    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:25.035673 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:25.035652    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:25:25.053957 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:25.053899    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podStartSLOduration=3.037991171 podStartE2EDuration="4.053884959s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.015893788s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:25:25.052832788 +0000 UTC m=+5093.085643126" watchObservedRunningTime="2026-07-30 18:25:25.053884959 +0000 UTC m=+5093.086695297"
Jul 30 18:25:26.038069 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:26.038022    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:25:31.042247 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:31.042217    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:25:31.042653 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:31.042626    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:25:41.043268 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:41.043221    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:25:51.043167 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:25:51.043122    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:26:01.043207 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:01.043126    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:26:11.043131 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:11.043085    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:26:21.042699 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:21.042657    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:26:31.042846 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:31.042806    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 30 18:26:41.043989 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.043959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:26:41.547831 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.547799    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"]
Jul 30 18:26:41.548044 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.548006    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container" containerID="cri-o://2d393010e9cb1edad1a1a3343350e436be59df57ecd16cd7cb9f5e8d8c304d58" gracePeriod=30
Jul 30 18:26:41.548132 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.548098    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kube-rbac-proxy" containerID="cri-o://0adbb5c4fde5534e97bba8b6ba5ab424d3816f808fed8aca14c6155a720c4204" gracePeriod=30
Jul 30 18:26:41.670349 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.670316    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"]
Jul 30 18:26:41.673495 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.673477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.676098 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.676073    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 30 18:26:41.676237 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.676136    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 30 18:26:41.676237 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.676150    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:26:41.682633 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.682609    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"]
Jul 30 18:26:41.861827 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.861728    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feee56e7-661e-492f-8254-b14f00203ac4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.861827 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.861814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feee56e7-661e-492f-8254-b14f00203ac4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.862023 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.861872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.862023 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.861927    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.862023 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.861952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42dw7\" (UniqueName: \"kubernetes.io/projected/feee56e7-661e-492f-8254-b14f00203ac4-kube-api-access-42dw7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.962646 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.962592    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.962851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.962671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.962851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.962701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42dw7\" (UniqueName: \"kubernetes.io/projected/feee56e7-661e-492f-8254-b14f00203ac4-kube-api-access-42dw7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.962851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.962744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feee56e7-661e-492f-8254-b14f00203ac4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.962851 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.962795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feee56e7-661e-492f-8254-b14f00203ac4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.963142 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.963118    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feee56e7-661e-492f-8254-b14f00203ac4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.963409 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.963388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.963409 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.963403    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.965334 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.965311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feee56e7-661e-492f-8254-b14f00203ac4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.971678 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.971656    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42dw7\" (UniqueName: \"kubernetes.io/projected/feee56e7-661e-492f-8254-b14f00203ac4-kube-api-access-42dw7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:41.984543 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:41.984523    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:42.117615 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:42.117589    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"]
Jul 30 18:26:42.120214 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:26:42.120171    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeee56e7_661e_492f_8254_b14f00203ac4.slice/crio-6f6c1435bc6410205dd2f5001563c19431406a6a1c8e75af231652131b870eae WatchSource:0}: Error finding container 6f6c1435bc6410205dd2f5001563c19431406a6a1c8e75af231652131b870eae: Status 404 returned error can't find the container with id 6f6c1435bc6410205dd2f5001563c19431406a6a1c8e75af231652131b870eae
Jul 30 18:26:42.324845 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:42.324811    2569 generic.go:356] "Generic (PLEG): container finished" podID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerID="0adbb5c4fde5534e97bba8b6ba5ab424d3816f808fed8aca14c6155a720c4204" exitCode=2
Jul 30 18:26:42.325059 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:42.324885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerDied","Data":"0adbb5c4fde5534e97bba8b6ba5ab424d3816f808fed8aca14c6155a720c4204"}
Jul 30 18:26:42.326405 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:42.326370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerStarted","Data":"3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3"}
Jul 30 18:26:42.326551 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:42.326420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerStarted","Data":"6f6c1435bc6410205dd2f5001563c19431406a6a1c8e75af231652131b870eae"}
Jul 30 18:26:43.331316 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:43.331282    2569 generic.go:356] "Generic (PLEG): container finished" podID="feee56e7-661e-492f-8254-b14f00203ac4" containerID="3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3" exitCode=0
Jul 30 18:26:43.331677 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:43.331347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerDied","Data":"3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3"}
Jul 30 18:26:44.342275 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:44.342218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerStarted","Data":"2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333"}
Jul 30 18:26:44.342690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:44.342324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerStarted","Data":"52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970"}
Jul 30 18:26:44.342690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:44.342355    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:44.342690 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:44.342526    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:44.343865 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:44.343841    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:26:44.362234 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:44.362178    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podStartSLOduration=2.310479098 podStartE2EDuration="3.362164929s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.051685831s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:26:44.360634281 +0000 UTC m=+5172.393444608" watchObservedRunningTime="2026-07-30 18:26:44.362164929 +0000 UTC m=+5172.394975267"
Jul 30 18:26:45.346672 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:45.346630    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:26:46.039169 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.039126    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jul 30 18:26:46.352425 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.352396    2569 generic.go:356] "Generic (PLEG): container finished" podID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerID="2d393010e9cb1edad1a1a3343350e436be59df57ecd16cd7cb9f5e8d8c304d58" exitCode=0
Jul 30 18:26:46.352790 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.352457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerDied","Data":"2d393010e9cb1edad1a1a3343350e436be59df57ecd16cd7cb9f5e8d8c304d58"}
Jul 30 18:26:46.396792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.396770    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:26:46.493566 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.493532    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls\") pod \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") "
Jul 30 18:26:46.493566 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.493566    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-rbac-proxy-sar-config\") pod \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") "
Jul 30 18:26:46.493834 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.493612    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kserve-provision-location\") pod \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") "
Jul 30 18:26:46.493834 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.493631    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9wrk\" (UniqueName: \"kubernetes.io/projected/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-api-access-c9wrk\") pod \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\" (UID: \"51fc2544-11e0-4b76-947b-6043a1bc8d7e\") "
Jul 30 18:26:46.493947 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.493919    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51fc2544-11e0-4b76-947b-6043a1bc8d7e" (UID: "51fc2544-11e0-4b76-947b-6043a1bc8d7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:26:46.494010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.493977    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51fc2544-11e0-4b76-947b-6043a1bc8d7e" (UID: "51fc2544-11e0-4b76-947b-6043a1bc8d7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:26:46.495745 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.495723    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51fc2544-11e0-4b76-947b-6043a1bc8d7e" (UID: "51fc2544-11e0-4b76-947b-6043a1bc8d7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:26:46.495854 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.495763    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-api-access-c9wrk" (OuterVolumeSpecName: "kube-api-access-c9wrk") pod "51fc2544-11e0-4b76-947b-6043a1bc8d7e" (UID: "51fc2544-11e0-4b76-947b-6043a1bc8d7e"). InnerVolumeSpecName "kube-api-access-c9wrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:26:46.594231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.594150    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:26:46.594231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.594185    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wrk\" (UniqueName: \"kubernetes.io/projected/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-api-access-c9wrk\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:26:46.594231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.594195    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51fc2544-11e0-4b76-947b-6043a1bc8d7e-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:26:46.594231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:46.594205    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51fc2544-11e0-4b76-947b-6043a1bc8d7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:26:47.358133 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.358095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr" event={"ID":"51fc2544-11e0-4b76-947b-6043a1bc8d7e","Type":"ContainerDied","Data":"ed57dfafbe04b903fa8a5ad8e7024c79245cd6a9cf7af71c6b5432b625138bef"}
Jul 30 18:26:47.358565 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.358145    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"
Jul 30 18:26:47.358565 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.358167    2569 scope.go:117] "RemoveContainer" containerID="0adbb5c4fde5534e97bba8b6ba5ab424d3816f808fed8aca14c6155a720c4204"
Jul 30 18:26:47.366271 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.366239    2569 scope.go:117] "RemoveContainer" containerID="2d393010e9cb1edad1a1a3343350e436be59df57ecd16cd7cb9f5e8d8c304d58"
Jul 30 18:26:47.372940 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.372922    2569 scope.go:117] "RemoveContainer" containerID="59cfb7e66fb9b51eb3e5ff951070c666501ce4bfd38491509e4e2f98a27397a0"
Jul 30 18:26:47.376957 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.376935    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"]
Jul 30 18:26:47.381814 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:47.381793    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5b6d75f8b7-tdbkr"]
Jul 30 18:26:48.651761 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:48.651732    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" path="/var/lib/kubelet/pods/51fc2544-11e0-4b76-947b-6043a1bc8d7e/volumes"
Jul 30 18:26:50.351395 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:50.351365    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:26:50.351857 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:26:50.351833    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:27:00.352035 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:27:00.351991    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:27:10.351968 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:27:10.351925    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:27:20.352292 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:27:20.352233    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:27:30.351867 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:27:30.351783    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:27:40.352560 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:27:40.352516    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:27:50.351838 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:27:50.351800    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 30 18:28:00.353344 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:00.353310    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:28:01.724264 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:01.724218    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"]
Jul 30 18:28:01.724760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:01.724502    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container" containerID="cri-o://52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970" gracePeriod=30
Jul 30 18:28:01.724760 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:01.724531    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kube-rbac-proxy" containerID="cri-o://2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333" gracePeriod=30
Jul 30 18:28:02.640210 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.640170    2569 generic.go:356] "Generic (PLEG): container finished" podID="feee56e7-661e-492f-8254-b14f00203ac4" containerID="2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333" exitCode=2
Jul 30 18:28:02.640418 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.640246    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerDied","Data":"2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333"}
Jul 30 18:28:02.786710 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.786669    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"]
Jul 30 18:28:02.787176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787086    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="storage-initializer"
Jul 30 18:28:02.787176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787107    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="storage-initializer"
Jul 30 18:28:02.787176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787123    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container"
Jul 30 18:28:02.787176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787133    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container"
Jul 30 18:28:02.787176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787145    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kube-rbac-proxy"
Jul 30 18:28:02.787176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787154    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kube-rbac-proxy"
Jul 30 18:28:02.787541 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787219    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kube-rbac-proxy"
Jul 30 18:28:02.787541 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.787234    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fc2544-11e0-4b76-947b-6043a1bc8d7e" containerName="kserve-container"
Jul 30 18:28:02.790474 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.790449    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.792802 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.792783    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 30 18:28:02.792930 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.792855    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 30 18:28:02.799604 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.799579    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"]
Jul 30 18:28:02.890696 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.890601    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a8384e-fe9f-40d2-a429-abaa28e85932-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.890696 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.890645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a8384e-fe9f-40d2-a429-abaa28e85932-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.890962 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.890712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.890962 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.890731    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96sf\" (UniqueName: \"kubernetes.io/projected/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-api-access-c96sf\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.991793 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.991752    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a8384e-fe9f-40d2-a429-abaa28e85932-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.991793 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.991794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a8384e-fe9f-40d2-a429-abaa28e85932-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.992084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.991841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.992084 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.991860    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96sf\" (UniqueName: \"kubernetes.io/projected/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-api-access-c96sf\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.992198 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.992185    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a8384e-fe9f-40d2-a429-abaa28e85932-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.992515 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.992494    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:02.994364 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:02.994342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a8384e-fe9f-40d2-a429-abaa28e85932-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:03.001191 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:03.001167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96sf\" (UniqueName: \"kubernetes.io/projected/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-api-access-c96sf\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:03.102245 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:03.102205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:03.228617 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:03.228592    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"]
Jul 30 18:28:03.230948 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:28:03.230921    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a8384e_fe9f_40d2_a429_abaa28e85932.slice/crio-be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14 WatchSource:0}: Error finding container be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14: Status 404 returned error can't find the container with id be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14
Jul 30 18:28:03.645758 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:03.645720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" event={"ID":"b4a8384e-fe9f-40d2-a429-abaa28e85932","Type":"ContainerStarted","Data":"69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc"}
Jul 30 18:28:03.645930 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:03.645794    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" event={"ID":"b4a8384e-fe9f-40d2-a429-abaa28e85932","Type":"ContainerStarted","Data":"be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14"}
Jul 30 18:28:05.347235 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:05.347194    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jul 30 18:28:06.465861 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.465837    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:28:06.515231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515200    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feee56e7-661e-492f-8254-b14f00203ac4-kserve-provision-location\") pod \"feee56e7-661e-492f-8254-b14f00203ac4\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") "
Jul 30 18:28:06.515407 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515243    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-kube-rbac-proxy-sar-config\") pod \"feee56e7-661e-492f-8254-b14f00203ac4\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") "
Jul 30 18:28:06.515407 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515299    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-cabundle-cert\") pod \"feee56e7-661e-492f-8254-b14f00203ac4\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") "
Jul 30 18:28:06.515407 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515328    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feee56e7-661e-492f-8254-b14f00203ac4-proxy-tls\") pod \"feee56e7-661e-492f-8254-b14f00203ac4\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") "
Jul 30 18:28:06.515407 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515360    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42dw7\" (UniqueName: \"kubernetes.io/projected/feee56e7-661e-492f-8254-b14f00203ac4-kube-api-access-42dw7\") pod \"feee56e7-661e-492f-8254-b14f00203ac4\" (UID: \"feee56e7-661e-492f-8254-b14f00203ac4\") "
Jul 30 18:28:06.515648 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515555    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feee56e7-661e-492f-8254-b14f00203ac4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "feee56e7-661e-492f-8254-b14f00203ac4" (UID: "feee56e7-661e-492f-8254-b14f00203ac4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:28:06.515698 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515649    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "feee56e7-661e-492f-8254-b14f00203ac4" (UID: "feee56e7-661e-492f-8254-b14f00203ac4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:28:06.515736 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.515704    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "feee56e7-661e-492f-8254-b14f00203ac4" (UID: "feee56e7-661e-492f-8254-b14f00203ac4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:28:06.517423 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.517383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feee56e7-661e-492f-8254-b14f00203ac4-kube-api-access-42dw7" (OuterVolumeSpecName: "kube-api-access-42dw7") pod "feee56e7-661e-492f-8254-b14f00203ac4" (UID: "feee56e7-661e-492f-8254-b14f00203ac4"). InnerVolumeSpecName "kube-api-access-42dw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:28:06.517423 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.517415    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feee56e7-661e-492f-8254-b14f00203ac4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feee56e7-661e-492f-8254-b14f00203ac4" (UID: "feee56e7-661e-492f-8254-b14f00203ac4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:28:06.615899 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.615873    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42dw7\" (UniqueName: \"kubernetes.io/projected/feee56e7-661e-492f-8254-b14f00203ac4-kube-api-access-42dw7\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:06.615899 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.615900    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feee56e7-661e-492f-8254-b14f00203ac4-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:06.616027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.615910    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:06.616027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.615921    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feee56e7-661e-492f-8254-b14f00203ac4-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:06.616027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.615930    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feee56e7-661e-492f-8254-b14f00203ac4-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:06.663781 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.663758    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/0.log"
Jul 30 18:28:06.663888 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.663796    2569 generic.go:356] "Generic (PLEG): container finished" podID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerID="69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc" exitCode=1
Jul 30 18:28:06.663888 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.663871    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" event={"ID":"b4a8384e-fe9f-40d2-a429-abaa28e85932","Type":"ContainerDied","Data":"69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc"}
Jul 30 18:28:06.665730 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.665703    2569 generic.go:356] "Generic (PLEG): container finished" podID="feee56e7-661e-492f-8254-b14f00203ac4" containerID="52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970" exitCode=0
Jul 30 18:28:06.665847 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.665740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerDied","Data":"52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970"}
Jul 30 18:28:06.665847 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.665760    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"
Jul 30 18:28:06.665847 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.665791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm" event={"ID":"feee56e7-661e-492f-8254-b14f00203ac4","Type":"ContainerDied","Data":"6f6c1435bc6410205dd2f5001563c19431406a6a1c8e75af231652131b870eae"}
Jul 30 18:28:06.665847 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.665829    2569 scope.go:117] "RemoveContainer" containerID="2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333"
Jul 30 18:28:06.673830 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.673807    2569 scope.go:117] "RemoveContainer" containerID="52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970"
Jul 30 18:28:06.681317 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.681287    2569 scope.go:117] "RemoveContainer" containerID="3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3"
Jul 30 18:28:06.696367 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.696346    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"]
Jul 30 18:28:06.700173 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.700157    2569 scope.go:117] "RemoveContainer" containerID="2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333"
Jul 30 18:28:06.700498 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:06.700478    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333\": container with ID starting with 2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333 not found: ID does not exist" containerID="2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333"
Jul 30 18:28:06.700579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.700508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333"} err="failed to get container status \"2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333\": rpc error: code = NotFound desc = could not find container \"2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333\": container with ID starting with 2a4098c3ba3b03e7993504e4a0ae5f7ebb92a000c0675b0571c7b82427e4d333 not found: ID does not exist"
Jul 30 18:28:06.700579 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.700528    2569 scope.go:117] "RemoveContainer" containerID="52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970"
Jul 30 18:28:06.700759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.700737    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5594fd785f-rjzxm"]
Jul 30 18:28:06.700848 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:06.700809    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970\": container with ID starting with 52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970 not found: ID does not exist" containerID="52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970"
Jul 30 18:28:06.700902 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.700841    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970"} err="failed to get container status \"52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970\": rpc error: code = NotFound desc = could not find container \"52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970\": container with ID starting with 52bc042f203e5d38cc05b2f1bc9046738e3a83bacc944cc34d338ead1ea44970 not found: ID does not exist"
Jul 30 18:28:06.700902 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.700865    2569 scope.go:117] "RemoveContainer" containerID="3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3"
Jul 30 18:28:06.701105 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:06.701085    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3\": container with ID starting with 3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3 not found: ID does not exist" containerID="3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3"
Jul 30 18:28:06.701165 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:06.701114    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3"} err="failed to get container status \"3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3\": rpc error: code = NotFound desc = could not find container \"3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3\": container with ID starting with 3e378a1d0d5af4e4d4f62f99e6cbf913328d00b4639c834fbb12a8b6fbb362b3 not found: ID does not exist"
Jul 30 18:28:07.672457 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:07.672430    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/0.log"
Jul 30 18:28:07.672837 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:07.672488    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" event={"ID":"b4a8384e-fe9f-40d2-a429-abaa28e85932","Type":"ContainerStarted","Data":"5e40118a75325ceb10b87c5c4a062993f8afebd714afe46ea423f5b55960c30c"}
Jul 30 18:28:08.651626 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:08.651585    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feee56e7-661e-492f-8254-b14f00203ac4" path="/var/lib/kubelet/pods/feee56e7-661e-492f-8254-b14f00203ac4/volumes"
Jul 30 18:28:11.687905 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.687876    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/1.log"
Jul 30 18:28:11.688349 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.688334    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/0.log"
Jul 30 18:28:11.688405 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.688368    2569 generic.go:356] "Generic (PLEG): container finished" podID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerID="5e40118a75325ceb10b87c5c4a062993f8afebd714afe46ea423f5b55960c30c" exitCode=1
Jul 30 18:28:11.688444 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.688429    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" event={"ID":"b4a8384e-fe9f-40d2-a429-abaa28e85932","Type":"ContainerDied","Data":"5e40118a75325ceb10b87c5c4a062993f8afebd714afe46ea423f5b55960c30c"}
Jul 30 18:28:11.688497 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.688486    2569 scope.go:117] "RemoveContainer" containerID="69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc"
Jul 30 18:28:11.688915 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.688897    2569 scope.go:117] "RemoveContainer" containerID="69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc"
Jul 30 18:28:11.698709 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:11.698682    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_kserve-ci-e2e-test(b4a8384e-fe9f-40d2-a429-abaa28e85932)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932"
Jul 30 18:28:11.698815 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:11.698790    2569 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_kserve-ci-e2e-test_b4a8384e-fe9f-40d2-a429-abaa28e85932_0 in pod sandbox be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14 from index: no such id: '69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc'" containerID="69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc"
Jul 30 18:28:11.698869 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:11.698823    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_kserve-ci-e2e-test_b4a8384e-fe9f-40d2-a429-abaa28e85932_0 in pod sandbox be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14 from index: no such id: '69770a543290a2a14baa546d8a2a5008adcb5f417b98bd468e846ff48ac50ddc'"
Jul 30 18:28:12.692994 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.692965    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/1.log"
Jul 30 18:28:12.787143 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.787108    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"]
Jul 30 18:28:12.910962 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.910940    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/1.log"
Jul 30 18:28:12.914575 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.914556    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:12.958244 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958154    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a8384e-fe9f-40d2-a429-abaa28e85932-kserve-provision-location\") pod \"b4a8384e-fe9f-40d2-a429-abaa28e85932\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") "
Jul 30 18:28:12.958244 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958240    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-rbac-proxy-sar-config\") pod \"b4a8384e-fe9f-40d2-a429-abaa28e85932\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") "
Jul 30 18:28:12.958466 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958280    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c96sf\" (UniqueName: \"kubernetes.io/projected/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-api-access-c96sf\") pod \"b4a8384e-fe9f-40d2-a429-abaa28e85932\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") "
Jul 30 18:28:12.958466 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958319    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a8384e-fe9f-40d2-a429-abaa28e85932-proxy-tls\") pod \"b4a8384e-fe9f-40d2-a429-abaa28e85932\" (UID: \"b4a8384e-fe9f-40d2-a429-abaa28e85932\") "
Jul 30 18:28:12.958466 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958394    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a8384e-fe9f-40d2-a429-abaa28e85932-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4a8384e-fe9f-40d2-a429-abaa28e85932" (UID: "b4a8384e-fe9f-40d2-a429-abaa28e85932"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:28:12.958643 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958516    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4a8384e-fe9f-40d2-a429-abaa28e85932-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:12.958643 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.958570    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4a8384e-fe9f-40d2-a429-abaa28e85932" (UID: "b4a8384e-fe9f-40d2-a429-abaa28e85932"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:28:12.960306 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.960280    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-api-access-c96sf" (OuterVolumeSpecName: "kube-api-access-c96sf") pod "b4a8384e-fe9f-40d2-a429-abaa28e85932" (UID: "b4a8384e-fe9f-40d2-a429-abaa28e85932"). InnerVolumeSpecName "kube-api-access-c96sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:28:12.960404 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:12.960366    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a8384e-fe9f-40d2-a429-abaa28e85932-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4a8384e-fe9f-40d2-a429-abaa28e85932" (UID: "b4a8384e-fe9f-40d2-a429-abaa28e85932"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:28:13.058876 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.058841    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:13.058876 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.058870    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96sf\" (UniqueName: \"kubernetes.io/projected/b4a8384e-fe9f-40d2-a429-abaa28e85932-kube-api-access-c96sf\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:13.058876 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.058882    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4a8384e-fe9f-40d2-a429-abaa28e85932-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:28:13.698189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.698159    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf_b4a8384e-fe9f-40d2-a429-abaa28e85932/storage-initializer/1.log"
Jul 30 18:28:13.698644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.698274    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"
Jul 30 18:28:13.698644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.698290    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf" event={"ID":"b4a8384e-fe9f-40d2-a429-abaa28e85932","Type":"ContainerDied","Data":"be8758d40db0e131fc5308a2d2e978642a8fc0705793b98c76c546ef27101f14"}
Jul 30 18:28:13.698644 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.698354    2569 scope.go:117] "RemoveContainer" containerID="5e40118a75325ceb10b87c5c4a062993f8afebd714afe46ea423f5b55960c30c"
Jul 30 18:28:13.741004 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.740972    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"]
Jul 30 18:28:13.746145 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.746123    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-85dc5ddc9-7vkmf"]
Jul 30 18:28:13.913900 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.913860    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"]
Jul 30 18:28:13.914214 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914197    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kube-rbac-proxy"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914219    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kube-rbac-proxy"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914246    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerName="storage-initializer"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914270    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerName="storage-initializer"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914283    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914292    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914301    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerName="storage-initializer"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914310    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerName="storage-initializer"
Jul 30 18:28:13.914326 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914322    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="storage-initializer"
Jul 30 18:28:13.914727 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914330    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="storage-initializer"
Jul 30 18:28:13.914727 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914396    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerName="storage-initializer"
Jul 30 18:28:13.914727 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914406    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" containerName="storage-initializer"
Jul 30 18:28:13.914727 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914416    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kube-rbac-proxy"
Jul 30 18:28:13.914727 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.914427    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="feee56e7-661e-492f-8254-b14f00203ac4" containerName="kserve-container"
Jul 30 18:28:13.919310 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.919289    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:13.921905 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.921873    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 30 18:28:13.922026 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.921921    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-plw5t\""
Jul 30 18:28:13.922026 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.921970    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 30 18:28:13.922026 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.922010    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 30 18:28:13.922204 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.922040    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 30 18:28:13.922204 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.922176    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 30 18:28:13.922428 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.922414    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:28:13.927807 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.927786    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"]
Jul 30 18:28:13.963362 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.963276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbcp\" (UniqueName: \"kubernetes.io/projected/2c360436-a55b-403a-b972-eca5d55098c2-kube-api-access-hdbcp\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:13.963509 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.963353    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:13.963509 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.963413    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c360436-a55b-403a-b972-eca5d55098c2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:13.963509 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.963442    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:13.963509 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:13.963472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.063786 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.063742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.063990 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.063802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbcp\" (UniqueName: \"kubernetes.io/projected/2c360436-a55b-403a-b972-eca5d55098c2-kube-api-access-hdbcp\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.063990 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:14.063906    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:28:14.063990 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.063972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.063990 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:28:14.063982    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls podName:2c360436-a55b-403a-b972-eca5d55098c2 nodeName:}" failed. No retries permitted until 2026-07-30 18:28:14.56396512 +0000 UTC m=+5262.596775434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" (UID: "2c360436-a55b-403a-b972-eca5d55098c2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:28:14.064209 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.064026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c360436-a55b-403a-b972-eca5d55098c2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.064209 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.064070    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.064505 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.064483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c360436-a55b-403a-b972-eca5d55098c2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.064662 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.064645    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.064707 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.064674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.073045 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.073026    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbcp\" (UniqueName: \"kubernetes.io/projected/2c360436-a55b-403a-b972-eca5d55098c2-kube-api-access-hdbcp\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.567272 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.567210    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.569796 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.569768    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.652580 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.652547    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a8384e-fe9f-40d2-a429-abaa28e85932" path="/var/lib/kubelet/pods/b4a8384e-fe9f-40d2-a429-abaa28e85932/volumes"
Jul 30 18:28:14.830015 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.829933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:14.962110 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:14.962080    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"]
Jul 30 18:28:14.964952 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:28:14.964923    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c360436_a55b_403a_b972_eca5d55098c2.slice/crio-2598ed49dea9ea17ca4d481dd68e6cc2a9114644873ab97daec3b9dfc0425f28 WatchSource:0}: Error finding container 2598ed49dea9ea17ca4d481dd68e6cc2a9114644873ab97daec3b9dfc0425f28: Status 404 returned error can't find the container with id 2598ed49dea9ea17ca4d481dd68e6cc2a9114644873ab97daec3b9dfc0425f28
Jul 30 18:28:15.709076 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:15.709037    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerStarted","Data":"91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651"}
Jul 30 18:28:15.709231 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:15.709107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerStarted","Data":"2598ed49dea9ea17ca4d481dd68e6cc2a9114644873ab97daec3b9dfc0425f28"}
Jul 30 18:28:16.714773 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:16.714741    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c360436-a55b-403a-b972-eca5d55098c2" containerID="91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651" exitCode=0
Jul 30 18:28:16.715141 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:16.714809    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerDied","Data":"91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651"}
Jul 30 18:28:17.721842 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:17.721804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerStarted","Data":"4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72"}
Jul 30 18:28:17.722374 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:17.721899    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:17.722374 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:17.721921    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerStarted","Data":"f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab"}
Jul 30 18:28:17.743502 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:17.743453    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podStartSLOduration=3.710137488 podStartE2EDuration="4.743440296s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.033302808s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:28:17.742387387 +0000 UTC m=+5265.775197726" watchObservedRunningTime="2026-07-30 18:28:17.743440296 +0000 UTC m=+5265.776250632"
Jul 30 18:28:18.726303 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:18.726249    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:18.727569 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:18.727538    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:28:19.729839 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:19.729798    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:28:24.734710 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:24.734674    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:28:24.735297 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:24.735245    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:28:34.735292 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:34.735231    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:28:44.735880 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:44.735831    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:28:54.735972 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:28:54.735890    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:29:04.735270 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:04.735211    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:29:14.736101 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:14.736056    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:29:24.735786 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:24.735746    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 30 18:29:33.647943 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:33.647913    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:29:33.930482 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:33.930398    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"]
Jul 30 18:29:34.077448 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:34.077403    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:34.077657 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:34.077496    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls podName:2c360436-a55b-403a-b972-eca5d55098c2 nodeName:}" failed. No retries permitted until 2026-07-30 18:29:34.577472943 +0000 UTC m=+5342.610283263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" (UID: "2c360436-a55b-403a-b972-eca5d55098c2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:34.579698 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:34.579661    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:34.579891 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:34.579732    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls podName:2c360436-a55b-403a-b972-eca5d55098c2 nodeName:}" failed. No retries permitted until 2026-07-30 18:29:35.579716174 +0000 UTC m=+5343.612526490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" (UID: "2c360436-a55b-403a-b972-eca5d55098c2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:34.995400 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:34.995364    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"]
Jul 30 18:29:34.998702 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:34.998678    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.001365 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.001344    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 30 18:29:35.001521 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.001369    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 30 18:29:35.001808 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.001763    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container" containerID="cri-o://f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab" gracePeriod=30
Jul 30 18:29:35.001916 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.001831    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kube-rbac-proxy" containerID="cri-o://4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72" gracePeriod=30
Jul 30 18:29:35.009592 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.009566    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"]
Jul 30 18:29:35.083173 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.083133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56618f3b-0a79-4479-81e8-8134c66adca8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.083173 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.083180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56618f3b-0a79-4479-81e8-8134c66adca8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.083418 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.083208    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhps\" (UniqueName: \"kubernetes.io/projected/56618f3b-0a79-4479-81e8-8134c66adca8-kube-api-access-bvhps\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.083418 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.083243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.184088 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.184045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.184319 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.184119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56618f3b-0a79-4479-81e8-8134c66adca8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.184319 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.184148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56618f3b-0a79-4479-81e8-8134c66adca8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.184319 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.184168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhps\" (UniqueName: \"kubernetes.io/projected/56618f3b-0a79-4479-81e8-8134c66adca8-kube-api-access-bvhps\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.184319 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:35.184213    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 30 18:29:35.184319 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:35.184310    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls podName:56618f3b-0a79-4479-81e8-8134c66adca8 nodeName:}" failed. No retries permitted until 2026-07-30 18:29:35.684288639 +0000 UTC m=+5343.717098974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" (UID: "56618f3b-0a79-4479-81e8-8134c66adca8") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 30 18:29:35.184673 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.184652    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56618f3b-0a79-4479-81e8-8134c66adca8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.184863 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.184845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56618f3b-0a79-4479-81e8-8134c66adca8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.194503 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.194476    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhps\" (UniqueName: \"kubernetes.io/projected/56618f3b-0a79-4479-81e8-8134c66adca8-kube-api-access-bvhps\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.587347 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:35.587310    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:35.587536 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:35.587381    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls podName:2c360436-a55b-403a-b972-eca5d55098c2 nodeName:}" failed. No retries permitted until 2026-07-30 18:29:37.587365289 +0000 UTC m=+5345.620175604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" (UID: "2c360436-a55b-403a-b972-eca5d55098c2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:35.688287 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.688237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.690711 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.690687    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:35.916861 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:35.916772    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:36.008228 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:36.008196    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c360436-a55b-403a-b972-eca5d55098c2" containerID="4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72" exitCode=2
Jul 30 18:29:36.008605 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:36.008275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerDied","Data":"4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72"}
Jul 30 18:29:36.045092 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:36.045059    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"]
Jul 30 18:29:36.048357 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:29:36.048332    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56618f3b_0a79_4479_81e8_8134c66adca8.slice/crio-e554a021027cb1a2db950f659ac25bc99add9481d5d95d391383138a4ee9aa08 WatchSource:0}: Error finding container e554a021027cb1a2db950f659ac25bc99add9481d5d95d391383138a4ee9aa08: Status 404 returned error can't find the container with id e554a021027cb1a2db950f659ac25bc99add9481d5d95d391383138a4ee9aa08
Jul 30 18:29:37.013902 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:37.013860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" event={"ID":"56618f3b-0a79-4479-81e8-8134c66adca8","Type":"ContainerStarted","Data":"8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21"}
Jul 30 18:29:37.014395 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:37.013976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" event={"ID":"56618f3b-0a79-4479-81e8-8134c66adca8","Type":"ContainerStarted","Data":"e554a021027cb1a2db950f659ac25bc99add9481d5d95d391383138a4ee9aa08"}
Jul 30 18:29:37.604482 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:37.604443    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:37.604672 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:37.604525    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls podName:2c360436-a55b-403a-b972-eca5d55098c2 nodeName:}" failed. No retries permitted until 2026-07-30 18:29:41.604510975 +0000 UTC m=+5349.637321291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" (UID: "2c360436-a55b-403a-b972-eca5d55098c2") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 30 18:29:39.642347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.642320    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:29:39.724646 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.724616    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-kube-rbac-proxy-sar-config\") pod \"2c360436-a55b-403a-b972-eca5d55098c2\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") "
Jul 30 18:29:39.724838 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.724654    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdbcp\" (UniqueName: \"kubernetes.io/projected/2c360436-a55b-403a-b972-eca5d55098c2-kube-api-access-hdbcp\") pod \"2c360436-a55b-403a-b972-eca5d55098c2\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") "
Jul 30 18:29:39.724838 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.724688    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls\") pod \"2c360436-a55b-403a-b972-eca5d55098c2\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") "
Jul 30 18:29:39.724838 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.724710    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-cabundle-cert\") pod \"2c360436-a55b-403a-b972-eca5d55098c2\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") "
Jul 30 18:29:39.724838 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.724748    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c360436-a55b-403a-b972-eca5d55098c2-kserve-provision-location\") pod \"2c360436-a55b-403a-b972-eca5d55098c2\" (UID: \"2c360436-a55b-403a-b972-eca5d55098c2\") "
Jul 30 18:29:39.725123 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.725078    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c360436-a55b-403a-b972-eca5d55098c2" (UID: "2c360436-a55b-403a-b972-eca5d55098c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:29:39.725123 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.725113    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c360436-a55b-403a-b972-eca5d55098c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c360436-a55b-403a-b972-eca5d55098c2" (UID: "2c360436-a55b-403a-b972-eca5d55098c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:29:39.725367 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.725168    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c360436-a55b-403a-b972-eca5d55098c2" (UID: "2c360436-a55b-403a-b972-eca5d55098c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:29:39.726830 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.726803    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c360436-a55b-403a-b972-eca5d55098c2" (UID: "2c360436-a55b-403a-b972-eca5d55098c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:29:39.726830 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.726819    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c360436-a55b-403a-b972-eca5d55098c2-kube-api-access-hdbcp" (OuterVolumeSpecName: "kube-api-access-hdbcp") pod "2c360436-a55b-403a-b972-eca5d55098c2" (UID: "2c360436-a55b-403a-b972-eca5d55098c2"). InnerVolumeSpecName "kube-api-access-hdbcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:29:39.825247 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.825215    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c360436-a55b-403a-b972-eca5d55098c2-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:39.825247 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.825246    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:39.825247 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.825272    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbcp\" (UniqueName: \"kubernetes.io/projected/2c360436-a55b-403a-b972-eca5d55098c2-kube-api-access-hdbcp\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:39.825515 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.825286    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c360436-a55b-403a-b972-eca5d55098c2-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:39.825515 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:39.825299    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c360436-a55b-403a-b972-eca5d55098c2-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:40.028096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.027999    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c360436-a55b-403a-b972-eca5d55098c2" containerID="f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab" exitCode=0
Jul 30 18:29:40.028096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.028053    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"
Jul 30 18:29:40.028096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.028063    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerDied","Data":"f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab"}
Jul 30 18:29:40.028399 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.028122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn" event={"ID":"2c360436-a55b-403a-b972-eca5d55098c2","Type":"ContainerDied","Data":"2598ed49dea9ea17ca4d481dd68e6cc2a9114644873ab97daec3b9dfc0425f28"}
Jul 30 18:29:40.028399 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.028161    2569 scope.go:117] "RemoveContainer" containerID="4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72"
Jul 30 18:29:40.036595 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.036574    2569 scope.go:117] "RemoveContainer" containerID="f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab"
Jul 30 18:29:40.043424 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.043405    2569 scope.go:117] "RemoveContainer" containerID="91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651"
Jul 30 18:29:40.050422 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.050402    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"]
Jul 30 18:29:40.050751 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.050740    2569 scope.go:117] "RemoveContainer" containerID="4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72"
Jul 30 18:29:40.050990 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:40.050975    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72\": container with ID starting with 4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72 not found: ID does not exist" containerID="4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72"
Jul 30 18:29:40.051028 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.050998    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72"} err="failed to get container status \"4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72\": rpc error: code = NotFound desc = could not find container \"4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72\": container with ID starting with 4bb361217c3e50d859c7b80825f0f0d0d633b71e273599c355c7e0379679ea72 not found: ID does not exist"
Jul 30 18:29:40.051028 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.051015    2569 scope.go:117] "RemoveContainer" containerID="f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab"
Jul 30 18:29:40.051236 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:40.051221    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab\": container with ID starting with f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab not found: ID does not exist" containerID="f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab"
Jul 30 18:29:40.051398 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.051241    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab"} err="failed to get container status \"f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab\": rpc error: code = NotFound desc = could not find container \"f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab\": container with ID starting with f53bd67551b6b9a1a54408add2eda6af6b6ba4ae42748b6cc9e09da7782f92ab not found: ID does not exist"
Jul 30 18:29:40.051398 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.051284    2569 scope.go:117] "RemoveContainer" containerID="91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651"
Jul 30 18:29:40.051502 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:40.051481    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651\": container with ID starting with 91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651 not found: ID does not exist" containerID="91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651"
Jul 30 18:29:40.051539 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.051508    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651"} err="failed to get container status \"91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651\": rpc error: code = NotFound desc = could not find container \"91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651\": container with ID starting with 91f9ed57e76214340521a85593597313b2cc57dd51f8371fd4dc2b42202ae651 not found: ID does not exist"
Jul 30 18:29:40.054572 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.054547    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-67df58d45f-qbxkn"]
Jul 30 18:29:40.651960 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:40.651922    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c360436-a55b-403a-b972-eca5d55098c2" path="/var/lib/kubelet/pods/2c360436-a55b-403a-b972-eca5d55098c2/volumes"
Jul 30 18:29:43.041873 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:43.041795    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs_56618f3b-0a79-4479-81e8-8134c66adca8/storage-initializer/0.log"
Jul 30 18:29:43.041873 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:43.041837    2569 generic.go:356] "Generic (PLEG): container finished" podID="56618f3b-0a79-4479-81e8-8134c66adca8" containerID="8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21" exitCode=1
Jul 30 18:29:43.042301 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:43.041920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" event={"ID":"56618f3b-0a79-4479-81e8-8134c66adca8","Type":"ContainerDied","Data":"8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21"}
Jul 30 18:29:44.047201 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:44.047174    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs_56618f3b-0a79-4479-81e8-8134c66adca8/storage-initializer/0.log"
Jul 30 18:29:44.047601 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:44.047276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" event={"ID":"56618f3b-0a79-4479-81e8-8134c66adca8","Type":"ContainerStarted","Data":"bebc59fe27d2db05bc5164fef56b96c7ef12880bb0e8f514ac20c22c5983eacb"}
Jul 30 18:29:45.000266 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:45.000220    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"]
Jul 30 18:29:45.050988 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:45.050951    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer" containerID="cri-o://bebc59fe27d2db05bc5164fef56b96c7ef12880bb0e8f514ac20c22c5983eacb" gracePeriod=30
Jul 30 18:29:46.105956 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.105919    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"]
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106206    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="storage-initializer"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106218    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="storage-initializer"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106234    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kube-rbac-proxy"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106239    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kube-rbac-proxy"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106247    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106269    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106320    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kserve-container"
Jul 30 18:29:46.106347 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.106327    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c360436-a55b-403a-b972-eca5d55098c2" containerName="kube-rbac-proxy"
Jul 30 18:29:46.109349 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.109328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.112022 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.112002    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 30 18:29:46.112152 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.112127    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 30 18:29:46.112152 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.112139    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 30 18:29:46.119637 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.119616    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"]
Jul 30 18:29:46.175592 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.175551    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.175778 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.175596    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56nfk\" (UniqueName: \"kubernetes.io/projected/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-api-access-56nfk\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.175778 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.175671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.175778 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.175727    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.175901 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.175781    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c397c5-fdea-4722-a2ba-aeba298e9936-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.276791 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.276750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.276791 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.276796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56nfk\" (UniqueName: \"kubernetes.io/projected/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-api-access-56nfk\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.277054 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.276847    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.277054 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.276876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.277054 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:46.276913    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 30 18:29:46.277054 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.276925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c397c5-fdea-4722-a2ba-aeba298e9936-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.277054 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:46.277003    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls podName:66c397c5-fdea-4722-a2ba-aeba298e9936 nodeName:}" failed. No retries permitted until 2026-07-30 18:29:46.776980169 +0000 UTC m=+5354.809790504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" (UID: "66c397c5-fdea-4722-a2ba-aeba298e9936") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 30 18:29:46.277416 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.277397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c397c5-fdea-4722-a2ba-aeba298e9936-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.277641 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.277623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.277682 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.277665    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.285730 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.285712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56nfk\" (UniqueName: \"kubernetes.io/projected/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-api-access-56nfk\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.781975 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.781939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:46.784507 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:46.784487    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:47.019304 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:47.019243    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:47.148667 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:47.148634    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"]
Jul 30 18:29:47.151861 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:29:47.151819    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c397c5_fdea_4722_a2ba_aeba298e9936.slice/crio-79a89a89af96ce7d43eb2fd759252e1c278d46a6dc33f9a6e5ad28808aa40847 WatchSource:0}: Error finding container 79a89a89af96ce7d43eb2fd759252e1c278d46a6dc33f9a6e5ad28808aa40847: Status 404 returned error can't find the container with id 79a89a89af96ce7d43eb2fd759252e1c278d46a6dc33f9a6e5ad28808aa40847
Jul 30 18:29:48.064011 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:48.063971    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerStarted","Data":"48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6"}
Jul 30 18:29:48.064225 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:48.064031    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerStarted","Data":"79a89a89af96ce7d43eb2fd759252e1c278d46a6dc33f9a6e5ad28808aa40847"}
Jul 30 18:29:49.069669 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:49.069632    2569 generic.go:356] "Generic (PLEG): container finished" podID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerID="48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6" exitCode=0
Jul 30 18:29:49.070058 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:49.069729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerDied","Data":"48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6"}
Jul 30 18:29:50.076480 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.076443    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerStarted","Data":"a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4"}
Jul 30 18:29:50.077010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.076510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerStarted","Data":"14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c"}
Jul 30 18:29:50.077010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.076655    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:50.077010 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.076844    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:50.078269 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.078230    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs_56618f3b-0a79-4479-81e8-8134c66adca8/storage-initializer/1.log"
Jul 30 18:29:50.078554 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.078528    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:29:50.078712 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.078696    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs_56618f3b-0a79-4479-81e8-8134c66adca8/storage-initializer/0.log"
Jul 30 18:29:50.078772 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.078732    2569 generic.go:356] "Generic (PLEG): container finished" podID="56618f3b-0a79-4479-81e8-8134c66adca8" containerID="bebc59fe27d2db05bc5164fef56b96c7ef12880bb0e8f514ac20c22c5983eacb" exitCode=1
Jul 30 18:29:50.078772 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.078761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" event={"ID":"56618f3b-0a79-4479-81e8-8134c66adca8","Type":"ContainerDied","Data":"bebc59fe27d2db05bc5164fef56b96c7ef12880bb0e8f514ac20c22c5983eacb"}
Jul 30 18:29:50.078857 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.078811    2569 scope.go:117] "RemoveContainer" containerID="8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21"
Jul 30 18:29:50.096829 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.096780    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podStartSLOduration=3.043248183 podStartE2EDuration="4.096762666s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.053514483s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:29:50.094176319 +0000 UTC m=+5358.126986656" watchObservedRunningTime="2026-07-30 18:29:50.096762666 +0000 UTC m=+5358.129573004"
Jul 30 18:29:50.098146 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.098018    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs_56618f3b-0a79-4479-81e8-8134c66adca8/storage-initializer/1.log"
Jul 30 18:29:50.098316 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:29:50.098291    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21\": container with ID starting with 8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21 not found: ID does not exist" containerID="8e98a4f0ca8f19174bc7c821d4ff686e090f2cc511d8425f8f2a07e6850eab21"
Jul 30 18:29:50.101529 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.101512    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:50.209910 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.209878    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56618f3b-0a79-4479-81e8-8134c66adca8-kube-rbac-proxy-sar-config\") pod \"56618f3b-0a79-4479-81e8-8134c66adca8\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") "
Jul 30 18:29:50.210108 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.209939    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56618f3b-0a79-4479-81e8-8134c66adca8-kserve-provision-location\") pod \"56618f3b-0a79-4479-81e8-8134c66adca8\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") "
Jul 30 18:29:50.210108 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.209974    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvhps\" (UniqueName: \"kubernetes.io/projected/56618f3b-0a79-4479-81e8-8134c66adca8-kube-api-access-bvhps\") pod \"56618f3b-0a79-4479-81e8-8134c66adca8\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") "
Jul 30 18:29:50.210108 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.210012    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls\") pod \"56618f3b-0a79-4479-81e8-8134c66adca8\" (UID: \"56618f3b-0a79-4479-81e8-8134c66adca8\") "
Jul 30 18:29:50.210318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.210206    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56618f3b-0a79-4479-81e8-8134c66adca8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56618f3b-0a79-4479-81e8-8134c66adca8" (UID: "56618f3b-0a79-4479-81e8-8134c66adca8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:29:50.210318 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.210297    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56618f3b-0a79-4479-81e8-8134c66adca8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56618f3b-0a79-4479-81e8-8134c66adca8" (UID: "56618f3b-0a79-4479-81e8-8134c66adca8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:29:50.212097 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.212072    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56618f3b-0a79-4479-81e8-8134c66adca8" (UID: "56618f3b-0a79-4479-81e8-8134c66adca8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:29:50.212174 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.212144    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56618f3b-0a79-4479-81e8-8134c66adca8-kube-api-access-bvhps" (OuterVolumeSpecName: "kube-api-access-bvhps") pod "56618f3b-0a79-4479-81e8-8134c66adca8" (UID: "56618f3b-0a79-4479-81e8-8134c66adca8"). InnerVolumeSpecName "kube-api-access-bvhps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:29:50.310924 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.310868    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56618f3b-0a79-4479-81e8-8134c66adca8-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:50.310924 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.310915    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhps\" (UniqueName: \"kubernetes.io/projected/56618f3b-0a79-4479-81e8-8134c66adca8-kube-api-access-bvhps\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:50.310924 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.310926    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56618f3b-0a79-4479-81e8-8134c66adca8-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:50.310924 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:50.310937    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56618f3b-0a79-4479-81e8-8134c66adca8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:29:51.083319 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.083291    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs_56618f3b-0a79-4479-81e8-8134c66adca8/storage-initializer/1.log"
Jul 30 18:29:51.083804 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.083394    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"
Jul 30 18:29:51.083804 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.083428    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs" event={"ID":"56618f3b-0a79-4479-81e8-8134c66adca8","Type":"ContainerDied","Data":"e554a021027cb1a2db950f659ac25bc99add9481d5d95d391383138a4ee9aa08"}
Jul 30 18:29:51.083804 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.083495    2569 scope.go:117] "RemoveContainer" containerID="bebc59fe27d2db05bc5164fef56b96c7ef12880bb0e8f514ac20c22c5983eacb"
Jul 30 18:29:51.084169 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.084138    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:29:51.115919 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.115888    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"]
Jul 30 18:29:51.121286 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:51.121242    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-c79fdb4b7-p9cxs"]
Jul 30 18:29:52.651716 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:52.651685    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" path="/var/lib/kubelet/pods/56618f3b-0a79-4479-81e8-8134c66adca8/volumes"
Jul 30 18:29:56.087901 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:56.087869    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:29:56.088401 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:29:56.088375    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:30:06.088976 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:30:06.088936    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:30:16.088432 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:30:16.088394    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:30:26.088829 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:30:26.088732    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:30:36.088357 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:30:36.088314    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:30:46.089128 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:30:46.089084    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:30:56.088590 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:30:56.088549    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 30 18:31:06.089456 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:06.089421    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:31:16.136366 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:16.136331    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"]
Jul 30 18:31:16.136755 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:16.136607    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container" containerID="cri-o://14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c" gracePeriod=30
Jul 30 18:31:16.136755 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:16.136660    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kube-rbac-proxy" containerID="cri-o://a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4" gracePeriod=30
Jul 30 18:31:16.396047 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:16.395957    2569 generic.go:356] "Generic (PLEG): container finished" podID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerID="a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4" exitCode=2
Jul 30 18:31:16.396047 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:16.396028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerDied","Data":"a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4"}
Jul 30 18:31:17.261377 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261338    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"]
Jul 30 18:31:17.261874 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261602    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer"
Jul 30 18:31:17.261874 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261612    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer"
Jul 30 18:31:17.261874 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261677    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer"
Jul 30 18:31:17.261874 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261693    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer"
Jul 30 18:31:17.261874 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261738    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer"
Jul 30 18:31:17.261874 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.261744    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56618f3b-0a79-4479-81e8-8134c66adca8" containerName="storage-initializer"
Jul 30 18:31:17.268983 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.268950    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.271527 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.271502    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 30 18:31:17.271662 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.271504    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 30 18:31:17.274897 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.274855    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"]
Jul 30 18:31:17.374305 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.374272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.374459 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.374327    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.374459 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.374370    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsbg\" (UniqueName: \"kubernetes.io/projected/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-api-access-vdsbg\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.374459 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.374445    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.475021 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.474985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.475193 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.475026    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.475193 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.475136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsbg\" (UniqueName: \"kubernetes.io/projected/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-api-access-vdsbg\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.475193 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:17.475154    2569 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 30 18:31:17.475193 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.475168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.475411 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:17.475275    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls podName:7f2eca8a-7b68-433a-8998-ef0ec85f17e2 nodeName:}" failed. No retries permitted until 2026-07-30 18:31:17.975235598 +0000 UTC m=+5446.008045933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" (UID: "7f2eca8a-7b68-433a-8998-ef0ec85f17e2") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 30 18:31:17.475567 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.475549    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.475872 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.475856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.485822 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.485796    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsbg\" (UniqueName: \"kubernetes.io/projected/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-api-access-vdsbg\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.979515 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.979460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:17.981880 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:17.981860    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:18.180978 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:18.180922    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:18.312842 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:18.312811    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"]
Jul 30 18:31:18.316729 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:31:18.316700    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2eca8a_7b68_433a_8998_ef0ec85f17e2.slice/crio-558548139bd57fc30f86bb3880ed66821eed44751850b8f80d8767d69d0deba2 WatchSource:0}: Error finding container 558548139bd57fc30f86bb3880ed66821eed44751850b8f80d8767d69d0deba2: Status 404 returned error can't find the container with id 558548139bd57fc30f86bb3880ed66821eed44751850b8f80d8767d69d0deba2
Jul 30 18:31:18.406945 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:18.406910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" event={"ID":"7f2eca8a-7b68-433a-8998-ef0ec85f17e2","Type":"ContainerStarted","Data":"29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba"}
Jul 30 18:31:18.407072 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:18.406969    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" event={"ID":"7f2eca8a-7b68-433a-8998-ef0ec85f17e2","Type":"ContainerStarted","Data":"558548139bd57fc30f86bb3880ed66821eed44751850b8f80d8767d69d0deba2"}
Jul 30 18:31:20.784580 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.784552    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:31:20.904272 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904157    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56nfk\" (UniqueName: \"kubernetes.io/projected/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-api-access-56nfk\") pod \"66c397c5-fdea-4722-a2ba-aeba298e9936\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") "
Jul 30 18:31:20.904272 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904210    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-rbac-proxy-sar-config\") pod \"66c397c5-fdea-4722-a2ba-aeba298e9936\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") "
Jul 30 18:31:20.904272 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904243    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-cabundle-cert\") pod \"66c397c5-fdea-4722-a2ba-aeba298e9936\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") "
Jul 30 18:31:20.904581 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904383    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c397c5-fdea-4722-a2ba-aeba298e9936-kserve-provision-location\") pod \"66c397c5-fdea-4722-a2ba-aeba298e9936\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") "
Jul 30 18:31:20.904581 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904463    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls\") pod \"66c397c5-fdea-4722-a2ba-aeba298e9936\" (UID: \"66c397c5-fdea-4722-a2ba-aeba298e9936\") "
Jul 30 18:31:20.904674 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904599    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66c397c5-fdea-4722-a2ba-aeba298e9936" (UID: "66c397c5-fdea-4722-a2ba-aeba298e9936"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:31:20.904674 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904629    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66c397c5-fdea-4722-a2ba-aeba298e9936" (UID: "66c397c5-fdea-4722-a2ba-aeba298e9936"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:31:20.904754 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904673    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c397c5-fdea-4722-a2ba-aeba298e9936-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66c397c5-fdea-4722-a2ba-aeba298e9936" (UID: "66c397c5-fdea-4722-a2ba-aeba298e9936"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:31:20.904754 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904693    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:20.904754 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.904711    2569 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c397c5-fdea-4722-a2ba-aeba298e9936-cabundle-cert\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:20.906365 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.906336    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-api-access-56nfk" (OuterVolumeSpecName: "kube-api-access-56nfk") pod "66c397c5-fdea-4722-a2ba-aeba298e9936" (UID: "66c397c5-fdea-4722-a2ba-aeba298e9936"). InnerVolumeSpecName "kube-api-access-56nfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:31:20.906475 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:20.906430    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66c397c5-fdea-4722-a2ba-aeba298e9936" (UID: "66c397c5-fdea-4722-a2ba-aeba298e9936"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:31:21.005658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.005620    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c397c5-fdea-4722-a2ba-aeba298e9936-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:21.005658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.005650    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56nfk\" (UniqueName: \"kubernetes.io/projected/66c397c5-fdea-4722-a2ba-aeba298e9936-kube-api-access-56nfk\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:21.005658 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.005660    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c397c5-fdea-4722-a2ba-aeba298e9936-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:21.418852 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.418819    2569 generic.go:356] "Generic (PLEG): container finished" podID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerID="14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c" exitCode=0
Jul 30 18:31:21.419027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.418885    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"
Jul 30 18:31:21.419027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.418898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerDied","Data":"14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c"}
Jul 30 18:31:21.419027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.418961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z" event={"ID":"66c397c5-fdea-4722-a2ba-aeba298e9936","Type":"ContainerDied","Data":"79a89a89af96ce7d43eb2fd759252e1c278d46a6dc33f9a6e5ad28808aa40847"}
Jul 30 18:31:21.419027 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.418989    2569 scope.go:117] "RemoveContainer" containerID="a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4"
Jul 30 18:31:21.427200 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.427184    2569 scope.go:117] "RemoveContainer" containerID="14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c"
Jul 30 18:31:21.434214 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.434196    2569 scope.go:117] "RemoveContainer" containerID="48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6"
Jul 30 18:31:21.440882 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.440861    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"]
Jul 30 18:31:21.441400 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.441386    2569 scope.go:117] "RemoveContainer" containerID="a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4"
Jul 30 18:31:21.441643 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:21.441627    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4\": container with ID starting with a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4 not found: ID does not exist" containerID="a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4"
Jul 30 18:31:21.441695 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.441650    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4"} err="failed to get container status \"a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4\": rpc error: code = NotFound desc = could not find container \"a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4\": container with ID starting with a637b54c5cab1bae9e889cf76f18c9d8bffb7925222454caf1e3694e9879dac4 not found: ID does not exist"
Jul 30 18:31:21.441695 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.441664    2569 scope.go:117] "RemoveContainer" containerID="14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c"
Jul 30 18:31:21.441887 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:21.441869    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c\": container with ID starting with 14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c not found: ID does not exist" containerID="14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c"
Jul 30 18:31:21.441931 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.441891    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c"} err="failed to get container status \"14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c\": rpc error: code = NotFound desc = could not find container \"14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c\": container with ID starting with 14a707b466f6ecafbb88e6ac350d9ebd596d69357028c363322b35dfb84fc59c not found: ID does not exist"
Jul 30 18:31:21.441931 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.441903    2569 scope.go:117] "RemoveContainer" containerID="48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6"
Jul 30 18:31:21.442094 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:21.442078    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6\": container with ID starting with 48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6 not found: ID does not exist" containerID="48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6"
Jul 30 18:31:21.442130 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.442097    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6"} err="failed to get container status \"48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6\": rpc error: code = NotFound desc = could not find container \"48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6\": container with ID starting with 48708efba7b0d6b29dc678e28b2513241f8513be150c8a2e8a0f65094642f8b6 not found: ID does not exist"
Jul 30 18:31:21.446149 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:21.446128    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c65f86bdd-ccp5z"]
Jul 30 18:31:22.425389 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:22.425360    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt_7f2eca8a-7b68-433a-8998-ef0ec85f17e2/storage-initializer/0.log"
Jul 30 18:31:22.425389 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:22.425392    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerID="29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba" exitCode=1
Jul 30 18:31:22.425842 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:22.425452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" event={"ID":"7f2eca8a-7b68-433a-8998-ef0ec85f17e2","Type":"ContainerDied","Data":"29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba"}
Jul 30 18:31:22.656654 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:22.653521    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" path="/var/lib/kubelet/pods/66c397c5-fdea-4722-a2ba-aeba298e9936/volumes"
Jul 30 18:31:23.431361 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:23.431332    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt_7f2eca8a-7b68-433a-8998-ef0ec85f17e2/storage-initializer/0.log"
Jul 30 18:31:23.431774 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:23.431416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" event={"ID":"7f2eca8a-7b68-433a-8998-ef0ec85f17e2","Type":"ContainerStarted","Data":"26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1"}
Jul 30 18:31:27.215068 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:27.215032    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"]
Jul 30 18:31:27.215492 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:27.215245    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer" containerID="cri-o://26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1" gracePeriod=30
Jul 30 18:31:28.957364 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:28.957342    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt_7f2eca8a-7b68-433a-8998-ef0ec85f17e2/storage-initializer/1.log"
Jul 30 18:31:28.957849 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:28.957830    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt_7f2eca8a-7b68-433a-8998-ef0ec85f17e2/storage-initializer/0.log"
Jul 30 18:31:28.960162 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:28.960142    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:29.072784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.072700    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls\") pod \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") "
Jul 30 18:31:29.072784 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.072745    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kserve-provision-location\") pod \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") "
Jul 30 18:31:29.073015 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.072788    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-rbac-proxy-sar-config\") pod \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") "
Jul 30 18:31:29.073015 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.072852    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdsbg\" (UniqueName: \"kubernetes.io/projected/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-api-access-vdsbg\") pod \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\" (UID: \"7f2eca8a-7b68-433a-8998-ef0ec85f17e2\") "
Jul 30 18:31:29.073117 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.073042    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f2eca8a-7b68-433a-8998-ef0ec85f17e2" (UID: "7f2eca8a-7b68-433a-8998-ef0ec85f17e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:31:29.073174 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.073144    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f2eca8a-7b68-433a-8998-ef0ec85f17e2" (UID: "7f2eca8a-7b68-433a-8998-ef0ec85f17e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 18:31:29.074910 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.074881    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f2eca8a-7b68-433a-8998-ef0ec85f17e2" (UID: "7f2eca8a-7b68-433a-8998-ef0ec85f17e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 18:31:29.075176 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.074921    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-api-access-vdsbg" (OuterVolumeSpecName: "kube-api-access-vdsbg") pod "7f2eca8a-7b68-433a-8998-ef0ec85f17e2" (UID: "7f2eca8a-7b68-433a-8998-ef0ec85f17e2"). InnerVolumeSpecName "kube-api-access-vdsbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:31:29.111006 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.110970    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mh28j/must-gather-dlnjw"]
Jul 30 18:31:29.111341 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111324    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kube-rbac-proxy"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111342    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kube-rbac-proxy"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111353    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111361    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111391    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111400    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111414    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="storage-initializer"
Jul 30 18:31:29.111435 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111422    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="storage-initializer"
Jul 30 18:31:29.111787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111486    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer"
Jul 30 18:31:29.111787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111498    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kserve-container"
Jul 30 18:31:29.111787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111507    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c397c5-fdea-4722-a2ba-aeba298e9936" containerName="kube-rbac-proxy"
Jul 30 18:31:29.111787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111586    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer"
Jul 30 18:31:29.111787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111595    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer"
Jul 30 18:31:29.111787 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.111651    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerName="storage-initializer"
Jul 30 18:31:29.114756 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.114735    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.117273 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.117238    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh28j\"/\"openshift-service-ca.crt\""
Jul 30 18:31:29.117388 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.117370    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mh28j\"/\"kube-root-ca.crt\""
Jul 30 18:31:29.117442 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.117417    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mh28j\"/\"default-dockercfg-scpxf\""
Jul 30 18:31:29.121168 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.121149    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh28j/must-gather-dlnjw"]
Jul 30 18:31:29.174189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.174158    2569 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-proxy-tls\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:29.174189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.174184    2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kserve-provision-location\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:29.174189 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.174194    2569 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:29.174436 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.174204    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsbg\" (UniqueName: \"kubernetes.io/projected/7f2eca8a-7b68-433a-8998-ef0ec85f17e2-kube-api-access-vdsbg\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:31:29.274792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.274751    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95xk\" (UniqueName: \"kubernetes.io/projected/dbddc1d7-bffc-483f-b5df-c99550e110d2-kube-api-access-d95xk\") pod \"must-gather-dlnjw\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") " pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.274792 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.274798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dbddc1d7-bffc-483f-b5df-c99550e110d2-must-gather-output\") pod \"must-gather-dlnjw\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") " pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.375222 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.375128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95xk\" (UniqueName: \"kubernetes.io/projected/dbddc1d7-bffc-483f-b5df-c99550e110d2-kube-api-access-d95xk\") pod \"must-gather-dlnjw\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") " pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.375222 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.375182    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dbddc1d7-bffc-483f-b5df-c99550e110d2-must-gather-output\") pod \"must-gather-dlnjw\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") " pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.375559 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.375538    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dbddc1d7-bffc-483f-b5df-c99550e110d2-must-gather-output\") pod \"must-gather-dlnjw\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") " pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.385332 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.385302    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95xk\" (UniqueName: \"kubernetes.io/projected/dbddc1d7-bffc-483f-b5df-c99550e110d2-kube-api-access-d95xk\") pod \"must-gather-dlnjw\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") " pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.441336 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.441310    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:31:29.453768 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.453741    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt_7f2eca8a-7b68-433a-8998-ef0ec85f17e2/storage-initializer/1.log"
Jul 30 18:31:29.454146 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.454132    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt_7f2eca8a-7b68-433a-8998-ef0ec85f17e2/storage-initializer/0.log"
Jul 30 18:31:29.454227 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.454168    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" containerID="26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1" exitCode=1
Jul 30 18:31:29.454319 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.454269    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"
Jul 30 18:31:29.454319 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.454279    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" event={"ID":"7f2eca8a-7b68-433a-8998-ef0ec85f17e2","Type":"ContainerDied","Data":"26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1"}
Jul 30 18:31:29.454431 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.454339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt" event={"ID":"7f2eca8a-7b68-433a-8998-ef0ec85f17e2","Type":"ContainerDied","Data":"558548139bd57fc30f86bb3880ed66821eed44751850b8f80d8767d69d0deba2"}
Jul 30 18:31:29.454431 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.454366    2569 scope.go:117] "RemoveContainer" containerID="26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1"
Jul 30 18:31:29.461930 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.461917    2569 scope.go:117] "RemoveContainer" containerID="29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba"
Jul 30 18:31:29.469731 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.469714    2569 scope.go:117] "RemoveContainer" containerID="26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1"
Jul 30 18:31:29.470005 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:29.469987    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1\": container with ID starting with 26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1 not found: ID does not exist" containerID="26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1"
Jul 30 18:31:29.470065 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.470013    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1"} err="failed to get container status \"26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1\": rpc error: code = NotFound desc = could not find container \"26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1\": container with ID starting with 26535ec867bf5c190f7ff697278bb110736c0ebd8ae0aad10a58025ac2f31fd1 not found: ID does not exist"
Jul 30 18:31:29.470065 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.470029    2569 scope.go:117] "RemoveContainer" containerID="29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba"
Jul 30 18:31:29.470345 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:31:29.470326    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba\": container with ID starting with 29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba not found: ID does not exist" containerID="29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba"
Jul 30 18:31:29.470438 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.470350    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba"} err="failed to get container status \"29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba\": rpc error: code = NotFound desc = could not find container \"29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba\": container with ID starting with 29bdd3624bed327cab3ab029b44e47fbb87e909bc0f00439c247ee4cfedf65ba not found: ID does not exist"
Jul 30 18:31:29.494930 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.494895    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"]
Jul 30 18:31:29.500537 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.500510    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-78d97f4874-mfkqt"]
Jul 30 18:31:29.568540 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.568486    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mh28j/must-gather-dlnjw"]
Jul 30 18:31:29.571056 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:31:29.571033    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbddc1d7_bffc_483f_b5df_c99550e110d2.slice/crio-5783748d8537501b45f4b84b7591285bb958c7cb99f7b18dcea600e1220ab0c9 WatchSource:0}: Error finding container 5783748d8537501b45f4b84b7591285bb958c7cb99f7b18dcea600e1220ab0c9: Status 404 returned error can't find the container with id 5783748d8537501b45f4b84b7591285bb958c7cb99f7b18dcea600e1220ab0c9
Jul 30 18:31:29.572691 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:29.572677    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 18:31:30.464378 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:30.464298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh28j/must-gather-dlnjw" event={"ID":"dbddc1d7-bffc-483f-b5df-c99550e110d2","Type":"ContainerStarted","Data":"5783748d8537501b45f4b84b7591285bb958c7cb99f7b18dcea600e1220ab0c9"}
Jul 30 18:31:30.653749 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:30.653653    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2eca8a-7b68-433a-8998-ef0ec85f17e2" path="/var/lib/kubelet/pods/7f2eca8a-7b68-433a-8998-ef0ec85f17e2/volumes"
Jul 30 18:31:36.501060 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:36.501025    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh28j/must-gather-dlnjw" event={"ID":"dbddc1d7-bffc-483f-b5df-c99550e110d2","Type":"ContainerStarted","Data":"ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f"}
Jul 30 18:31:36.501060 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:36.501072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh28j/must-gather-dlnjw" event={"ID":"dbddc1d7-bffc-483f-b5df-c99550e110d2","Type":"ContainerStarted","Data":"ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"}
Jul 30 18:31:36.518606 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:31:36.518554    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mh28j/must-gather-dlnjw" podStartSLOduration=1.329550566 podStartE2EDuration="7.518536237s" totalImagesPullingTime="6.188985671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 18:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:31:36.51727012 +0000 UTC m=+5464.550080455" watchObservedRunningTime="2026-07-30 18:31:36.518536237 +0000 UTC m=+5464.551346575"
Jul 30 18:32:02.602102 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:02.602066    2569 generic.go:356] "Generic (PLEG): container finished" podID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerID="ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e" exitCode=0
Jul 30 18:32:02.602534 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:02.602142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mh28j/must-gather-dlnjw" event={"ID":"dbddc1d7-bffc-483f-b5df-c99550e110d2","Type":"ContainerDied","Data":"ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"}
Jul 30 18:32:02.602534 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:02.602495    2569 scope.go:117] "RemoveContainer" containerID="ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"
Jul 30 18:32:03.149390 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:03.149354    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mh28j_must-gather-dlnjw_dbddc1d7-bffc-483f-b5df-c99550e110d2/gather/0.log"
Jul 30 18:32:07.843982 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:07.843953    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2w6gg_5962bc8b-6237-4100-830f-5a54e63e0f14/global-pull-secret-syncer/0.log"
Jul 30 18:32:08.108865 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.108772    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sjw84_d27bd079-abf6-40c3-a09b-eca4993d05ed/konnectivity-agent/0.log"
Jul 30 18:32:08.160239 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.160209    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-31.ec2.internal_ef84da010c241ebfac81f7378c666253/haproxy/0.log"
Jul 30 18:32:08.686061 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.686029    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-mh28j/must-gather-dlnjw"]
Jul 30 18:32:08.686244 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.686229    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-mh28j/must-gather-dlnjw" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="copy" containerID="cri-o://ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f" gracePeriod=2
Jul 30 18:32:08.691127 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.691092    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-mh28j/must-gather-dlnjw"]
Jul 30 18:32:08.909608 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.909588    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mh28j_must-gather-dlnjw_dbddc1d7-bffc-483f-b5df-c99550e110d2/copy/0.log"
Jul 30 18:32:08.911348 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:08.911332    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:32:08.913433 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:32:08.913405    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dlnjw\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mh28j\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:32:09.001882 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.001791    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d95xk\" (UniqueName: \"kubernetes.io/projected/dbddc1d7-bffc-483f-b5df-c99550e110d2-kube-api-access-d95xk\") pod \"dbddc1d7-bffc-483f-b5df-c99550e110d2\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") "
Jul 30 18:32:09.001882 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.001843    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dbddc1d7-bffc-483f-b5df-c99550e110d2-must-gather-output\") pod \"dbddc1d7-bffc-483f-b5df-c99550e110d2\" (UID: \"dbddc1d7-bffc-483f-b5df-c99550e110d2\") "
Jul 30 18:32:09.003203 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.003168    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbddc1d7-bffc-483f-b5df-c99550e110d2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "dbddc1d7-bffc-483f-b5df-c99550e110d2" (UID: "dbddc1d7-bffc-483f-b5df-c99550e110d2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 18:32:09.004057 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.004034    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbddc1d7-bffc-483f-b5df-c99550e110d2-kube-api-access-d95xk" (OuterVolumeSpecName: "kube-api-access-d95xk") pod "dbddc1d7-bffc-483f-b5df-c99550e110d2" (UID: "dbddc1d7-bffc-483f-b5df-c99550e110d2"). InnerVolumeSpecName "kube-api-access-d95xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 18:32:09.102530 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.102502    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d95xk\" (UniqueName: \"kubernetes.io/projected/dbddc1d7-bffc-483f-b5df-c99550e110d2-kube-api-access-d95xk\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:32:09.102530 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.102528    2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dbddc1d7-bffc-483f-b5df-c99550e110d2-must-gather-output\") on node \"ip-10-0-142-31.ec2.internal\" DevicePath \"\""
Jul 30 18:32:09.631921 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.631893    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mh28j_must-gather-dlnjw_dbddc1d7-bffc-483f-b5df-c99550e110d2/copy/0.log"
Jul 30 18:32:09.632264 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.632231    2569 generic.go:356] "Generic (PLEG): container finished" podID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerID="ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f" exitCode=143
Jul 30 18:32:09.632336 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.632291    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:32:09.632410 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.632289    2569 scope.go:117] "RemoveContainer" containerID="ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f"
Jul 30 18:32:09.634661 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:32:09.634635    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dlnjw\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mh28j\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:32:09.639943 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.639924    2569 scope.go:117] "RemoveContainer" containerID="ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"
Jul 30 18:32:09.642298 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:32:09.642272    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-dlnjw\" is forbidden: User \"system:node:ip-10-0-142-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mh28j\": no relationship found between node 'ip-10-0-142-31.ec2.internal' and this object" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" pod="openshift-must-gather-mh28j/must-gather-dlnjw"
Jul 30 18:32:09.651465 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.651448    2569 scope.go:117] "RemoveContainer" containerID="ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f"
Jul 30 18:32:09.651708 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:32:09.651691    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f\": container with ID starting with ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f not found: ID does not exist" containerID="ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f"
Jul 30 18:32:09.651752 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.651715    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f"} err="failed to get container status \"ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f\": rpc error: code = NotFound desc = could not find container \"ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f\": container with ID starting with ba8f9ef3945f8495c3bd1c8f82a476f46d0a0cd16e899ed99e2cbcbefb5ee61f not found: ID does not exist"
Jul 30 18:32:09.651752 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.651731    2569 scope.go:117] "RemoveContainer" containerID="ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"
Jul 30 18:32:09.651989 ip-10-0-142-31 kubenswrapper[2569]: E0730 18:32:09.651976    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e\": container with ID starting with ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e not found: ID does not exist" containerID="ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"
Jul 30 18:32:09.652035 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:09.651992    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e"} err="failed to get container status \"ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e\": rpc error: code = NotFound desc = could not find container \"ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e\": container with ID starting with ed10b4b4c92d727b07fb748c3e9dacd7a212f7baf5488ddcf7fc86b4059b167e not found: ID does not exist"
Jul 30 18:32:10.651409 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:10.651377    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" path="/var/lib/kubelet/pods/dbddc1d7-bffc-483f-b5df-c99550e110d2/volumes"
Jul 30 18:32:12.307891 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.307854    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dknvj_b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01/kube-state-metrics/0.log"
Jul 30 18:32:12.329453 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.329424    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dknvj_b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01/kube-rbac-proxy-main/0.log"
Jul 30 18:32:12.351910 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.351885    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dknvj_b17d84c4-fdb6-4fda-8e7d-ddaef1e5fd01/kube-rbac-proxy-self/0.log"
Jul 30 18:32:12.607211 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.607178    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hzkvn_22db70d2-fa87-4b62-99ff-38988b3e2e63/node-exporter/0.log"
Jul 30 18:32:12.632207 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.632180    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hzkvn_22db70d2-fa87-4b62-99ff-38988b3e2e63/kube-rbac-proxy/0.log"
Jul 30 18:32:12.661130 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.661102    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hzkvn_22db70d2-fa87-4b62-99ff-38988b3e2e63/init-textfile/0.log"
Jul 30 18:32:12.792781 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.792742    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/prometheus/0.log"
Jul 30 18:32:12.812767 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.812732    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/config-reloader/0.log"
Jul 30 18:32:12.835565 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.835545    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/thanos-sidecar/0.log"
Jul 30 18:32:12.859426 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.859358    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/kube-rbac-proxy-web/0.log"
Jul 30 18:32:12.881330 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.881306    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/kube-rbac-proxy/0.log"
Jul 30 18:32:12.902941 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.902914    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/kube-rbac-proxy-thanos/0.log"
Jul 30 18:32:12.924366 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:12.924339    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b30290a-45b2-459f-ab8d-2ff3647a442c/init-config-reloader/0.log"
Jul 30 18:32:13.000050 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:13.000019    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-ghqtm_00ae6fa6-b8e2-43ec-aba5-d436a0aedb48/prometheus-operator-admission-webhook/0.log"
Jul 30 18:32:16.951815 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.951783    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"]
Jul 30 18:32:16.952197 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.952035    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="copy"
Jul 30 18:32:16.952197 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.952046    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="copy"
Jul 30 18:32:16.952197 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.952063    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="gather"
Jul 30 18:32:16.952197 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.952068    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="gather"
Jul 30 18:32:16.952197 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.952116    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="copy"
Jul 30 18:32:16.952197 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.952125    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbddc1d7-bffc-483f-b5df-c99550e110d2" containerName="gather"
Jul 30 18:32:16.955824 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.955806    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:16.958844 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.958811    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mgfj9\"/\"openshift-service-ca.crt\""
Jul 30 18:32:16.958962 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.958851    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mgfj9\"/\"kube-root-ca.crt\""
Jul 30 18:32:16.959486 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.959472    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mgfj9\"/\"default-dockercfg-rxz6h\""
Jul 30 18:32:16.962183 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:16.962158    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"]
Jul 30 18:32:17.068139 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.068098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-proc\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.068139 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.068135    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-lib-modules\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.068413 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.068167    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94k2\" (UniqueName: \"kubernetes.io/projected/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-kube-api-access-n94k2\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.068413 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.068236    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-podres\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.068413 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.068300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-sys\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.130930 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.130903    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-689qd_0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299/dns/0.log"
Jul 30 18:32:17.153098 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.153069    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-689qd_0e7a4eed-fed3-4c08-9fb3-4f1ae6a4d299/kube-rbac-proxy/0.log"
Jul 30 18:32:17.169514 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169482    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-podres\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169514 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-sys\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169548    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-proc\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169613    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-sys\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169674    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-proc\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169732    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-lib-modules\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169759 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-podres\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169966 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94k2\" (UniqueName: \"kubernetes.io/projected/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-kube-api-access-n94k2\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.169966 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.169834    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-lib-modules\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.179264 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.179229    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94k2\" (UniqueName: \"kubernetes.io/projected/dbcf803b-3901-40dc-9a2f-d00c9bf9b213-kube-api-access-n94k2\") pod \"perf-node-gather-daemonset-rj9z5\" (UID: \"dbcf803b-3901-40dc-9a2f-d00c9bf9b213\") " pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.266428 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.266332    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.316507 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.316455    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gj2bd_33f1a6e1-def7-469b-ad6d-82e62ad3075b/dns-node-resolver/0.log"
Jul 30 18:32:17.394211 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.394176    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"]
Jul 30 18:32:17.397318 ip-10-0-142-31 kubenswrapper[2569]: W0730 18:32:17.397287    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddbcf803b_3901_40dc_9a2f_d00c9bf9b213.slice/crio-88fe52be733887cbe5369e34e49c3bfaedd9147cfc30de76701fc7ac57e0deb1 WatchSource:0}: Error finding container 88fe52be733887cbe5369e34e49c3bfaedd9147cfc30de76701fc7ac57e0deb1: Status 404 returned error can't find the container with id 88fe52be733887cbe5369e34e49c3bfaedd9147cfc30de76701fc7ac57e0deb1
Jul 30 18:32:17.663545 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.663508    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5" event={"ID":"dbcf803b-3901-40dc-9a2f-d00c9bf9b213","Type":"ContainerStarted","Data":"c15703b70fe0f51b40bb4d817f7fdfda24b64a8c6cf65151fbe2d0c04b94dbd3"}
Jul 30 18:32:17.663737 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.663597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5" event={"ID":"dbcf803b-3901-40dc-9a2f-d00c9bf9b213","Type":"ContainerStarted","Data":"88fe52be733887cbe5369e34e49c3bfaedd9147cfc30de76701fc7ac57e0deb1"}
Jul 30 18:32:17.663737 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.663647    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:17.681141 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.681088    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5" podStartSLOduration=1.6810740480000002 podStartE2EDuration="1.681074048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 18:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 18:32:17.679775825 +0000 UTC m=+5505.712586163" watchObservedRunningTime="2026-07-30 18:32:17.681074048 +0000 UTC m=+5505.713884421"
Jul 30 18:32:17.839097 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.839008    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d6d5bb4cc-566rf_2906046b-8db0-4b67-a26d-5dcaeaaeb057/registry/0.log"
Jul 30 18:32:17.874040 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:17.874008    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llp2c_6a918e3e-b901-4f22-b193-4d40e2964878/node-ca/0.log"
Jul 30 18:32:18.962224 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:18.962184    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8k7v2_d476091d-f0e7-48d1-a149-f4bf94cf817b/serve-healthcheck-canary/0.log"
Jul 30 18:32:19.415557 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:19.415466    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9rcbw_5964c201-0d3d-444d-b808-33a3aaf5522e/kube-rbac-proxy/0.log"
Jul 30 18:32:19.435679 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:19.435648    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9rcbw_5964c201-0d3d-444d-b808-33a3aaf5522e/exporter/0.log"
Jul 30 18:32:19.456942 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:19.456918    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9rcbw_5964c201-0d3d-444d-b808-33a3aaf5522e/extractor/0.log"
Jul 30 18:32:21.928096 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:21.928065    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-6cpm8_8484bfc2-ecb1-4344-9a98-a59b27ecd7e7/manager/0.log"
Jul 30 18:32:22.334442 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:22.334409    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cnszb_813e4df1-d0c3-4a69-ae4e-e756cd6268fe/s3-init/0.log"
Jul 30 18:32:22.352962 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:22.352941    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cnszb_813e4df1-d0c3-4a69-ae4e-e756cd6268fe/download-hf-model/0.log"
Jul 30 18:32:22.386477 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:22.386446    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-27ksm_0e95a463-493b-4bed-a984-9b2bdb35aa7c/s3-tls-init-custom/0.log"
Jul 30 18:32:22.402722 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:22.402696    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-27ksm_0e95a463-493b-4bed-a984-9b2bdb35aa7c/download-hf-model/0.log"
Jul 30 18:32:22.435334 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:22.435311    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-6hsnc_078beaf9-1570-422f-be88-588dfb174d2d/s3-tls-init-serving/0.log"
Jul 30 18:32:22.453109 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:22.453085    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-6hsnc_078beaf9-1570-422f-be88-588dfb174d2d/download-hf-model/0.log"
Jul 30 18:32:23.676159 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:23.676132    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mgfj9/perf-node-gather-daemonset-rj9z5"
Jul 30 18:32:28.098525 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.098501    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/kube-multus-additional-cni-plugins/0.log"
Jul 30 18:32:28.120903 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.120875    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/egress-router-binary-copy/0.log"
Jul 30 18:32:28.142574 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.142553    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/cni-plugins/0.log"
Jul 30 18:32:28.165124 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.165100    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/bond-cni-plugin/0.log"
Jul 30 18:32:28.186459 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.186438    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/routeoverride-cni/0.log"
Jul 30 18:32:28.214676 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.214648    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/whereabouts-cni-bincopy/0.log"
Jul 30 18:32:28.235577 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.235503    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-swrqn_072815d7-8920-4d28-9e01-95e3a9908d71/whereabouts-cni/0.log"
Jul 30 18:32:28.285116 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.285085    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hz896_3d02675c-5b7b-4e38-a9a3-6aff3bd89563/kube-multus/0.log"
Jul 30 18:32:28.423911 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.423880    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qfh2d_ffa93763-d018-4c80-aa34-83bc592bc8b8/network-metrics-daemon/0.log"
Jul 30 18:32:28.440600 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:28.440564    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qfh2d_ffa93763-d018-4c80-aa34-83bc592bc8b8/kube-rbac-proxy/0.log"
Jul 30 18:32:29.157220 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.157189    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/ovn-controller/0.log"
Jul 30 18:32:29.173954 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.173925    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/ovn-acl-logging/0.log"
Jul 30 18:32:29.196958 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.196933    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/kube-rbac-proxy-node/0.log"
Jul 30 18:32:29.220385 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.220354    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 30 18:32:29.243590 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.243562    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/northd/0.log"
Jul 30 18:32:29.263919 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.263885    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/nbdb/0.log"
Jul 30 18:32:29.284809 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.284779    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/sbdb/0.log"
Jul 30 18:32:29.363868 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:29.363829    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l9f99_42f582b9-f05b-4c3e-996f-eebb4118a05a/ovnkube-controller/0.log"
Jul 30 18:32:30.782705 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:30.782677    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5gzq2_9587b1cf-a314-4e21-b752-1cadd4877321/network-check-target-container/0.log"
Jul 30 18:32:31.746968 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:31.746937    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vhmwn_48054529-4ab8-4319-a00d-bfd1577bddde/iptables-alerter/0.log"
Jul 30 18:32:32.423004 ip-10-0-142-31 kubenswrapper[2569]: I0730 18:32:32.422977    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-c5rf9_b7e5b982-a2b4-439a-8352-3966c1439ce0/tuned/0.log"