Jul 28 17:08:35.952389 ip-10-0-143-61 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:08:35.952401 ip-10-0-143-61 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:08:35.952407 ip-10-0-143-61 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:08:35.952609 ip-10-0-143-61 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:08:47.382825 ip-10-0-143-61 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:08:47.382839 ip-10-0-143-61 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 83cc24ff295b40cfb1db30988d80b965 --
Jul 28 17:10:15.133569 ip-10-0-143-61 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:10:15.525150 ip-10-0-143-61 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:15.525150 ip-10-0-143-61 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:10:15.525150 ip-10-0-143-61 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:15.525150 ip-10-0-143-61 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:10:15.525150 ip-10-0-143-61 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:15.526062 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.525992    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:10:15.530830 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530813    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:15.530830 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530828    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:15.530830 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530833    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530837    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530841    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530845    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530849    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530853    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530856    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530859    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530863    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530867    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530871    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530876    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530880    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530884    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530890    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530896    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530900    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530906    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530910    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:15.531011 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530916    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530921    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530925    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530929    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530933    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530937    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530942    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530946    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530950    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530954    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530959    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530963    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530967    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530971    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530975    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530980    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530984    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530991    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530996    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.530999    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:15.531791 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531003    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531007    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531011    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531016    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531019    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531028    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531033    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531038    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531042    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531047    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531051    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531055    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531059    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531063    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531067    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531071    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531075    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531079    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531083    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:15.532621 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531087    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531091    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531095    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531100    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531104    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531108    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531112    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531116    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531121    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531125    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531130    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531135    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531139    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531143    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531147    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531151    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531155    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531159    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531164    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531168    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:15.533417 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531172    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531175    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531180    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531184    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531188    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531192    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531196    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531200    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531204    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531208    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531212    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531218    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531222    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531226    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531230    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531234    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531238    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531258    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.531263    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532425    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:15.534235 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532435    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532439    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532443    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532448    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532453    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532457    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532461    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532465    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532469    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532473    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532477    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532482    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532486    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532489    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532493    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532497    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532501    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532506    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532511    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532515    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:15.535092 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532519    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532524    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532528    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532532    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532536    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532540    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532544    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532548    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532552    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532556    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532560    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532564    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532568    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532572    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532576    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532579    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532584    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532592    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532597    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:15.535833 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532602    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532607    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532610    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532615    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532618    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532622    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532626    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532630    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532634    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532638    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532642    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532646    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532651    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532658    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532662    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532666    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532670    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532674    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532679    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532683    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:15.536449 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532689    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532695    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532699    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532703    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532708    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532712    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532716    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532720    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532724    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532728    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532733    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532737    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532741    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532745    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532749    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532753    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532757    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532761    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532765    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532768    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:15.537002 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532773    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532777    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532782    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532789    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532793    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532797    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532801    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532805    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532809    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532812    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532817    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532820    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532825    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532828    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532833    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532838    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532842    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532846    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.532850    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:15.537517 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533451    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533461    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533466    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533470    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533475    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533480    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533484    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533491    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533496    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533501    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533506    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533511    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533515    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533519    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533523    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533528    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533533    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533538    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533543    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533547    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:15.538070 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533551    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533556    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533560    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533565    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533569    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533574    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533578    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533583    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533587    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533591    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533603    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533608    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533613    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533617    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533621    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533625    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533630    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533642    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533647    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:15.538623 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533651    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533655    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533659    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533663    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533667    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533671    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533675    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533679    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533683    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533687    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533692    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533696    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533699    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533703    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533707    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533711    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533715    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533719    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533723    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533727    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:15.539064 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533731    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533735    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533739    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533745    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533750    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533754    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533762    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533767    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533772    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533776    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533780    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533792    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533796    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533802    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533806    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533810    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533814    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533818    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533822    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:15.539581 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533826    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533830    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533837    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533841    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533845    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533849    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533853    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533857    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533860    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533864    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533869    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533873    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533877    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533881    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533884    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533889    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533892    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533896    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533901    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533905    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.533909    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:15.540044 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534003    2571 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534013    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534026    2571 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534033    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534049    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534055    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534061    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534067    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534071    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534076    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534081    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534086    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534091    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534097    2571 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534102    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534106    2571 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534110    2571 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534115    2571 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534125    2571 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534129    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534135    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534140    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534144    2571 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534150    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:10:15.540599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534154    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534159    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534164    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534169    2571 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534174    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534178    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534183    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534188    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534194    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534198    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534203    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534207    2571 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534212    2571 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534224    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534233    2571 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534238    2571 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534259    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534265    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534269    2571 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534276    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534280    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534286    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534291    2571 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534295    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534300    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:10:15.541190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534305    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534309    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534314    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534318    2571 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534323    2571 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534335    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534340    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534345    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534349    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534354    2571 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534359    2571 flags.go:64] FLAG: --help="false"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534364    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534370    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534376    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534380    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534386    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534391    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534396    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534400    2571 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534405    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534409    2571 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534429    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534435    2571 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534439    2571 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:10:15.541809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534444    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534449    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534453    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534458    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534464    2571 flags.go:64] FLAG: --lock-file=""
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534468    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534492    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534498    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534508    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534513    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534518    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534522    2571 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534527    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534532    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534536    2571 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534540    2571 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534547    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534552    2571 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534558    2571 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534562    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534568    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534573    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534577    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534582    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534587    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:10:15.542381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534592    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534602    2571 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534607    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534611    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534616    2571 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534628    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534637    2571 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534641    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534646    2571 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534650    2571 flags.go:64] FLAG: --port="10250"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534655    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534659    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0308acd38356435e4"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534667    2571 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534671    2571 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534676    2571 flags.go:64] FLAG: --register-node="true"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534680    2571 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534686    2571 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534690    2571 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534694    2571 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534699    2571 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534705    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534710    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534715    2571 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534719    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534724    2571 flags.go:64] FLAG: --runonce="false"
Jul 28 17:10:15.542968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534728    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534733    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534739    2571 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534743    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534748    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534752    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534757    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534762    2571 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534767    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534771    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534776    2571 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534780    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534785    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534791    2571 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534795    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534804    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534808    2571 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534813    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534822    2571 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534829    2571 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534833    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534838    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534842    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534847    2571 flags.go:64] FLAG: --v="2"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534852    2571 flags.go:64] FLAG: --version="false"
Jul 28 17:10:15.543573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534858    2571 flags.go:64] FLAG: --vmodule=""
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534864    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.534869    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535075    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535082    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535087    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535091    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535095    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535099    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535105    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535110    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535114    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535117    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535122    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535128    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535135    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535139    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535144    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535148    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535152    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:15.544183 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535156    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535162    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535166    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535171    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535175    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535179    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535185    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535189    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535193    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535197    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535201    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535205    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535210    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535214    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535218    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535222    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535227    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535230    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535234    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535238    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:15.544704 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535258    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535264    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535269    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535273    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535277    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535281    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535286    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535290    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535294    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535298    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535302    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535306    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535311    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535315    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535320    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535324    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535328    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535332    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535338    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535342    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:15.545166 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535346    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535350    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535354    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535358    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535363    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535367    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535371    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535375    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535378    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535382    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535386    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535393    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535399    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535405    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535410    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535414    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535418    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535422    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535426    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:15.545654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535430    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535434    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535439    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535443    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535447    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535452    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535456    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535459    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535464    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535468    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535472    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535478    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535482    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535485    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535489    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535493    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535497    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535501    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535505    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535511    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:15.546101 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535518    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535522    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.535526    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.535533    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.542084    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.542098    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.542112    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.542122    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542182    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542187    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542191    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542193    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542196    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542199    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:15.546598 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542202    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542205    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542207    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542210    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542213    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542216    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542231    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542234    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542236    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542239    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542241    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542257    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542260    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542263    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542265    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542268    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542271    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542273    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542276    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542278    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:15.546939 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542281    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542283    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542286    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542289    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542292    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542295    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542297    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542300    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542302    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542305    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542307    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542310    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542312    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542315    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542317    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542320    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542323    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542325    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542328    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542336    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:15.547418 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542339    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542341    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542344    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542347    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542349    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542352    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542354    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542357    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542359    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542361    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542364    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542366    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542368    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542371    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542374    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542376    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542379    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542381    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542384    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542387    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:15.547895 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542390    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542392    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542395    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542398    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542400    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542403    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542405    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542408    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542410    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542412    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542416    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542420    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542428    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542430    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542433    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542435    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542438    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542440    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542442    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:15.548515 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542445    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542447    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542450    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542452    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542456    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542459    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542462    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542466    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542470    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542473    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542476    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542478    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542481    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542484    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:15.548966 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.542488    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542635    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542640    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542643    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542646    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542648    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542651    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542653    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542656    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542660    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542663    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542666    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542675    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542679    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542681    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542684    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542686    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542689    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542691    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:15.549360 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542693    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542696    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542698    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542700    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542703    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542705    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542708    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542711    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542713    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542715    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542718    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542720    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542723    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542725    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542728    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542730    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542732    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542735    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542737    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542739    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:15.549803 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542742    2571 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542744    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542746    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542749    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542751    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542754    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542762    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542765    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542767    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542770    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542772    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542774    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542777    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542780    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542782    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542784    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542787    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542789    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542792    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542794    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:15.550294 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542796    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542799    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542801    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542803    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542806    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542808    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542811    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542813    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542815    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542818    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542820    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542822    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542825    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542827    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542829    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542832    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542834    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542837    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542840    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:15.551090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542849    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542852    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542855    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542857    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542859    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542863    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542866    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542868    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542871    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542874    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542876    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542878    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542881    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542884    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542887    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542889    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542891    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542900    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542902    2571 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542904    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:15.551647 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542907    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:15.542909    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.542913    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.543023    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.546088    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.547636    2571 server.go:1019] "Starting client certificate rotation"
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.547745    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:15.552203 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.548869    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:15.552704 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.552684    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:15.554933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.554915    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:15.565284 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.565270    2571 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:10:15.567212 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.567200    2571 log.go:25] "Validated CRI v1 image API"
Jul 28 17:10:15.568053 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.568035    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:10:15.571340 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.571322    2571 fs.go:135] Filesystem UUIDs: map[33409eb2-6428-4766-9e90-6483ba3a5bcb:/dev/nvme0n1p4 3c827fac-330d-4ed8-bdb0-d7f5fb7bdc77:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:10:15.571401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.571338    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:10:15.576238 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.576132    2571 manager.go:216] Machine: {Timestamp:2026-07-28 17:10:15.575008088 +0000 UTC m=+0.335033708 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100827 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279fb16f4f46094b7fcacc7970fa86 SystemUUID:ec279fb1-6f4f-4609-4b7f-cacc7970fa86 BootID:83cc24ff-295b-40cf-b1db-30988d80b965 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b7:b3:0e:4a:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b7:b3:0e:4a:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:42:27:3b:1c:8c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:10:15.576238 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.576233    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:10:15.576361 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.576313    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:10:15.579021 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.578996    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:10:15.579152 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.579021    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:10:15.579722 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.579712    2571 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:10:15.579754 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.579724    2571 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:10:15.579754 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.579731    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:15.580215 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.580204    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:15.581414 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.581403    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:10:15.584700 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.584689    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:15.584839 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.584829    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:10:15.585559 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585550    2571 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:10:15.585597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585569    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:10:15.585597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585579    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:10:15.585597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585587    2571 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:10:15.585597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585597    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:10:15.585898 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585888    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:15.585933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.585906    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:15.587898 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.587882    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:10:15.589258 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589231    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:10:15.589314 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589274    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:10:15.589934 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589921    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:10:15.589964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589938    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:10:15.589964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589944    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:10:15.589964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589949    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:10:15.589964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589956    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:10:15.589964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589961    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:10:15.590096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589968    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:10:15.590096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589973    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:10:15.590096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589979    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:10:15.590096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.589990    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:10:15.590096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.590002    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:10:15.590096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.590007    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:10:15.590833 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.590821    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:10:15.590833 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.590834    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:10:15.593811 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.593785    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:15.597117 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.597099    2571 server.go:1268] "Started kubelet"
Jul 28 17:10:15.597216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.597154    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:10:15.597282 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.597224    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:10:15.597328 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.597308    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:10:15.597810 ip-10-0-143-61 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:10:15.599039 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.599019    2571 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:10:15.599138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.599081    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:10:15.599203 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.599133    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:10:15.599203 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.599133    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:10:15.599479 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.599461    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:10:15.603127 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.603094    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:10:15.607519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.607503    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:10:15.607519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.607516    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:15.607634 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.607592    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:15.608032 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608017    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:10:15.608149 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608020    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:10:15.608149 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608143    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:10:15.608307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608218    2571 factory.go:55] Registering systemd factory
Jul 28 17:10:15.608307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608232    2571 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:10:15.608307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608272    2571 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:10:15.608307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608280    2571 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:10:15.608518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608421    2571 factory.go:153] Registering CRI-O factory
Jul 28 17:10:15.608518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608436    2571 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:10:15.608518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608489    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:10:15.608518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608515    2571 factory.go:103] Registering Raw factory
Jul 28 17:10:15.608656 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608530    2571 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:10:15.609823 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.608920    2571 manager.go:349] Starting recovery of all containers
Jul 28 17:10:15.609931 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.608352    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-61.ec2.internal.18c683408ede31fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.ec2.internal,UID:ip-10-0-143-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.ec2.internal,},FirstTimestamp:2026-07-28 17:10:15.597052411 +0000 UTC m=+0.357078029,LastTimestamp:2026-07-28 17:10:15.597052411 +0000 UTC m=+0.357078029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.ec2.internal,}"
Jul 28 17:10:15.612566 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.612537    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:10:15.612645 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.612618    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:10:15.619076 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.619052    2571 manager.go:354] Recovery completed
Jul 28 17:10:15.624783 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.624764    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.626607 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.626590    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.626702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.626617    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.626702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.626630    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.627126 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627112    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:10:15.627171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627125    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.627226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627126    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:10:15.627226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627207    2571 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:15.627938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627922    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.627998 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627948    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.627998 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.627961    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.628752 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.628739    2571 policy_none.go:49] "None policy: Start"
Jul 28 17:10:15.628821 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.628758    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:10:15.628821 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.628772    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:10:15.629357 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.629297    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-61.ec2.internal.18c6834090a12941  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.ec2.internal,UID:ip-10-0-143-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.ec2.internal,},FirstTimestamp:2026-07-28 17:10:15.626606913 +0000 UTC m=+0.386632533,LastTimestamp:2026-07-28 17:10:15.626606913 +0000 UTC m=+0.386632533,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.ec2.internal,}"
Jul 28 17:10:15.629971 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.629959    2571 policy_none.go:47] "Start"
Jul 28 17:10:15.641263 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.641190    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-61.ec2.internal.18c6834090a169a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.ec2.internal,UID:ip-10-0-143-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-61.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.ec2.internal,},FirstTimestamp:2026-07-28 17:10:15.626623395 +0000 UTC m=+0.386649015,LastTimestamp:2026-07-28 17:10:15.626623395 +0000 UTC m=+0.386649015,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.ec2.internal,}"
Jul 28 17:10:15.641874 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.641856    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zshx"
Jul 28 17:10:15.649468 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.649450    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zshx"
Jul 28 17:10:15.650231 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.650170    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-61.ec2.internal.18c6834090a19a2b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-61.ec2.internal,UID:ip-10-0-143-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-61.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-61.ec2.internal,},FirstTimestamp:2026-07-28 17:10:15.626635819 +0000 UTC m=+0.386661441,LastTimestamp:2026-07-28 17:10:15.626635819 +0000 UTC m=+0.386661441,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-61.ec2.internal,}"
Jul 28 17:10:15.651836 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.651821    2571 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:10:15.651904 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.651856    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:10:15.651904 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.651868    2571 server.go:87] "Starting device plugin registration server"
Jul 28 17:10:15.652098 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.652087    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:10:15.652187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.652103    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:10:15.652322 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.652310    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:10:15.652385 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.652373    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:10:15.652385 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.652385    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:10:15.652872 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.652857    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:10:15.652948 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.652905    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:15.700980 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.700959    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:10:15.703012 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.702038    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:10:15.703012 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.702075    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:10:15.703012 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.702094    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:10:15.703012 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.702123    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:10:15.704845 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.704830    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:15.752418 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.752277    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.753114 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.753096    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.753197 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.753125    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.753197 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.753137    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.753197 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.753159    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.764404 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.764387    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.764468 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.764405    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-61.ec2.internal\": node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:15.795087 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.795047    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:15.803106 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.803083    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"]
Jul 28 17:10:15.803184 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.803135    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.803802 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.803789    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.803865 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.803813    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.803865 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.803827    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.806148 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806136    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.806260 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806235    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.806313 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806271    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.806732 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806718    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.806791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806744    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.806791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806745    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.806791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806759    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.806791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806769    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.806791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.806781    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.809046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809029    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.809130 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809055    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:15.809266 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809233    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1804774383ef499fd25a45e880ee305-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.ec2.internal\" (UID: \"a1804774383ef499fd25a45e880ee305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.809336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809280    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b5e167bbe3082ae5f67dd557ac7ea1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"2b5e167bbe3082ae5f67dd557ac7ea1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.809336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5e167bbe3082ae5f67dd557ac7ea1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"2b5e167bbe3082ae5f67dd557ac7ea1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.809697 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809680    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:15.809772 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809707    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:15.809772 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.809718    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:15.838726 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.838708    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-61.ec2.internal\" not found" node="ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.842827 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.842814    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-61.ec2.internal\" not found" node="ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.895609 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.895592    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:15.909509 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.909490    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b5e167bbe3082ae5f67dd557ac7ea1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"2b5e167bbe3082ae5f67dd557ac7ea1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.909565 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.909512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5e167bbe3082ae5f67dd557ac7ea1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"2b5e167bbe3082ae5f67dd557ac7ea1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.909565 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.909533    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1804774383ef499fd25a45e880ee305-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.ec2.internal\" (UID: \"a1804774383ef499fd25a45e880ee305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.909625 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.909566    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1804774383ef499fd25a45e880ee305-config\") pod \"kube-apiserver-proxy-ip-10-0-143-61.ec2.internal\" (UID: \"a1804774383ef499fd25a45e880ee305\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.909625 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.909588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b5e167bbe3082ae5f67dd557ac7ea1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"2b5e167bbe3082ae5f67dd557ac7ea1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.909625 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:15.909589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b5e167bbe3082ae5f67dd557ac7ea1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal\" (UID: \"2b5e167bbe3082ae5f67dd557ac7ea1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:15.996062 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:15.996043    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.096799 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.096761    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.141284 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.141266    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:16.146373 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.146287    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:16.197153 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.197133    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.297605 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.297576    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.398056 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.398016    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.498484 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.498466    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.546943 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.546924    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:10:16.547274 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.547022    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:16.599363 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.599346    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.653849 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.653789    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:05:15 +0000 UTC" deadline="2028-04-08 11:01:28.240912782 +0000 UTC"
Jul 28 17:10:16.653849 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.653817    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14873h51m11.587098598s"
Jul 28 17:10:16.679975 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.679950    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:16.681031 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.681017    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:16.681086 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.681050    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:16.681086 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.681061    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:16.699585 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.699562    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.753986 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:16.753954    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1804774383ef499fd25a45e880ee305.slice/crio-4eaabf3fba52caa448a4eaf50d10383e49148dd1069cc996cba2ae561cdc063c WatchSource:0}: Error finding container 4eaabf3fba52caa448a4eaf50d10383e49148dd1069cc996cba2ae561cdc063c: Status 404 returned error can't find the container with id 4eaabf3fba52caa448a4eaf50d10383e49148dd1069cc996cba2ae561cdc063c
Jul 28 17:10:16.754654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:16.754615    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e167bbe3082ae5f67dd557ac7ea1b.slice/crio-61be0e8bb10ca3da868f0689d6650be38005e41539ba5b86338cfa179b40f9a7 WatchSource:0}: Error finding container 61be0e8bb10ca3da868f0689d6650be38005e41539ba5b86338cfa179b40f9a7: Status 404 returned error can't find the container with id 61be0e8bb10ca3da868f0689d6650be38005e41539ba5b86338cfa179b40f9a7
Jul 28 17:10:16.760658 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.760642    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:10:16.800535 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:16.800500    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-61.ec2.internal\" not found"
Jul 28 17:10:16.835229 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.835210    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:16.907913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.907868    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:16.923588 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.923568    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:16.926623 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.926610    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal"
Jul 28 17:10:16.938905 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:16.938889    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:17.154565 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.154542    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:17.212233 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.212188    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:17.586491 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.586413    2571 apiserver.go:52] "Watching apiserver"
Jul 28 17:10:17.595146 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.595114    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:10:17.595536 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.595508    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hll47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal","openshift-multus/multus-4lvf4","openshift-multus/multus-additional-cni-plugins-hvthd","openshift-multus/network-metrics-daemon-z4x8n","kube-system/konnectivity-agent-5mkh5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d","openshift-image-registry/node-ca-h5b8n","openshift-network-diagnostics/network-check-target-22bpl","openshift-network-operator/iptables-alerter-96tjh","openshift-ovn-kubernetes/ovnkube-node-wl2tm","kube-system/global-pull-secret-syncer-5rqtb","kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w9m2f"]
Jul 28 17:10:17.598295 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.598269    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.601039 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.601016    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.601814 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.601752    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.601814 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.601761    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jhccw\""
Jul 28 17:10:17.602008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.601991    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:10:17.602247 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.602225    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.603069 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.603055    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.604280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.604014    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.604280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.604041    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:10:17.604624 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.604602    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k56zz\""
Jul 28 17:10:17.604851 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.604835    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.605633    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.605700    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.605939    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.605946    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.605965    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xxghb\""
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.606141    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.606341 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.606202    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.606743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.606520    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.606743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.606574    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:10:17.608182 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.608164    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:17.608291 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.608249    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:17.608497 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.608478    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:17.610388 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.610328    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:10:17.610527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.610508    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ngkq2\""
Jul 28 17:10:17.610599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.610567    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:10:17.611632 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.611593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.614283 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.614263    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qls8h\""
Jul 28 17:10:17.614588 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.614562    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:17.614684 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.614637    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:17.614844 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.614749    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:10:17.614844 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.614823    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:10:17.616889 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.616869    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.619409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.619529 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619432    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkmc\" (UniqueName: \"kubernetes.io/projected/df7598ba-6577-4948-9df7-ba4528ee76b9-kube-api-access-8rkmc\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.619529 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2882d280-bc58-4983-89f7-bfc3273fee8f-agent-certs\") pod \"konnectivity-agent-5mkh5\" (UID: \"2882d280-bc58-4983-89f7-bfc3273fee8f\") " pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.619529 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovnkube-script-lib\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.619702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619539    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:17.619702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619569    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.619702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619601    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-netns\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.619702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619635    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-cni-multus\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.619702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.619702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619703    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-registration-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.619920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-systemd-units\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.619920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619761    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovnkube-config\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.619920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619794    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovn-node-metrics-cert\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.619920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619829    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.619920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-ovn\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619925    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-hostroot\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619957    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-daemon-config\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.619990    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-env-overrides\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620028    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.620280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-etc-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620160    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/da004608-b719-4ddd-b8fa-e55f4cb34081-kube-api-access-2zv7b\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620195    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-os-release\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.620280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-var-lib-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zmr\" (UniqueName: \"kubernetes.io/projected/70c2aa93-4685-4fe5-a1c6-eecf73c43442-kube-api-access-76zmr\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620288    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-system-cni-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-socket-dir-parent\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-k8s-cni-cncf-io\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-multus-certs\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-systemd\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-node-log\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-os-release\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-kubelet\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620571    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-system-cni-dir\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-sys-fs\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-kubelet\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-conf-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620736    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-cni-bin\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.620771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620770    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2882d280-bc58-4983-89f7-bfc3273fee8f-konnectivity-ca\") pod \"konnectivity-agent-5mkh5\" (UID: \"2882d280-bc58-4983-89f7-bfc3273fee8f\") " pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620804    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-cni-bin\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-etc-kubernetes\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68t2\" (UniqueName: \"kubernetes.io/projected/a722293e-4534-4652-be28-8df3b206b801-kube-api-access-w68t2\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620903    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-cni-netd\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-cnibin\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.620968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bx56\" (UniqueName: \"kubernetes.io/projected/0b662c0c-d267-40aa-8c94-748caf5b4735-kube-api-access-8bx56\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-run-netns\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621038    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-cni-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-cnibin\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da004608-b719-4ddd-b8fa-e55f4cb34081-cni-binary-copy\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-socket-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.621166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621165    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-device-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.621796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-run-ovn-kubernetes\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.621796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.621796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621263    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-slash\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.621796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.621301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-log-socket\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.622519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.622498    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.623548 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.623059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.624649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.624478    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.624649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.624509    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.624649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.624578    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wdg2v\""
Jul 28 17:10:17.625545 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.625527    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.626510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.625558    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:10:17.626510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.626057    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q488b\""
Jul 28 17:10:17.626510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.625693    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.626510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.625674    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.626510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.626356    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.626510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.626418    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.626886 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.626860    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:10:17.626992 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.626920    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:17.627303 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.627249    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lx6v9\""
Jul 28 17:10:17.630518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.630500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.630734 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.630713    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:17.630923 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.630786    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.631060 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.630913    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fww8f\""
Jul 28 17:10:17.631905 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.631860    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.653490 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.653471    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bj594"
Jul 28 17:10:17.663322 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.663119    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bj594"
Jul 28 17:10:17.707026 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.706978    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" event={"ID":"2b5e167bbe3082ae5f67dd557ac7ea1b","Type":"ContainerStarted","Data":"61be0e8bb10ca3da868f0689d6650be38005e41539ba5b86338cfa179b40f9a7"}
Jul 28 17:10:17.709137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.709105    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal" event={"ID":"a1804774383ef499fd25a45e880ee305","Type":"ContainerStarted","Data":"4eaabf3fba52caa448a4eaf50d10383e49148dd1069cc996cba2ae561cdc063c"}
Jul 28 17:10:17.709314 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.709296    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:10:17.721689 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721662    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-os-release\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.721804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721700    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-var-lib-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.721804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zmr\" (UniqueName: \"kubernetes.io/projected/70c2aa93-4685-4fe5-a1c6-eecf73c43442-kube-api-access-76zmr\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.721804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-system-cni-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.721804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721768    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-socket-dir-parent\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.721804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-k8s-cni-cncf-io\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721806    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-os-release\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-var-lib-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721864    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-multus-certs\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-multus-certs\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysctl-d\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-system-cni-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.721932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-host\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722087    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-socket-dir-parent\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722130    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-k8s-cni-cncf-io\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-systemd\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-node-log\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722218    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-os-release\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722266    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-systemd\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-kubelet\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-kubelet\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-system-cni-dir\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722349    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-system-cni-dir\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-sys-fs\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-kubelet\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722391    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-node-log\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-conf-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-kubelet-config\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722444    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-sys-fs\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.722459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722458    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-sys\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722499    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-os-release\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722507    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-systemd\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722550    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-kubelet\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722555    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-conf-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-iptables-alerter-script\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722648    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-cni-bin\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722674    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2882d280-bc58-4983-89f7-bfc3273fee8f-konnectivity-ca\") pod \"konnectivity-agent-5mkh5\" (UID: \"2882d280-bc58-4983-89f7-bfc3273fee8f\") " pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722699    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-dbus\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722707    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysctl-conf\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722768    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-cni-bin\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722774    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-tmp-dir\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722804    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-cni-bin\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722831    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-etc-kubernetes\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.723308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68t2\" (UniqueName: \"kubernetes.io/projected/a722293e-4534-4652-be28-8df3b206b801-kube-api-access-w68t2\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722878    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-cni-bin\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722880    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysconfig\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-lib-modules\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-etc-kubernetes\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-cni-netd\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.722975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-cnibin\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bx56\" (UniqueName: \"kubernetes.io/projected/0b662c0c-d267-40aa-8c94-748caf5b4735-kube-api-access-8bx56\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723025    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-host-slash\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723061    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-run-netns\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-cni-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723114    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-cnibin\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723139    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da004608-b719-4ddd-b8fa-e55f4cb34081-cni-binary-copy\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723163    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-socket-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723187    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-device-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723215    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-modprobe-d\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-kubernetes\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.724248 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-run-ovn-kubernetes\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723311    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztbk\" (UniqueName: \"kubernetes.io/projected/a71ac77c-691c-4723-bc03-974cd9990fdb-kube-api-access-dztbk\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-slash\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723397    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-cnibin\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-log-socket\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723447    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-run-netns\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723395    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-cni-netd\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723477    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-run-ovn-kubernetes\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723356    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2882d280-bc58-4983-89f7-bfc3273fee8f-konnectivity-ca\") pod \"konnectivity-agent-5mkh5\" (UID: \"2882d280-bc58-4983-89f7-bfc3273fee8f\") " pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723539    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-slash\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723574    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-log-socket\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723605    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723635    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkmc\" (UniqueName: \"kubernetes.io/projected/df7598ba-6577-4948-9df7-ba4528ee76b9-kube-api-access-8rkmc\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2882d280-bc58-4983-89f7-bfc3273fee8f-agent-certs\") pod \"konnectivity-agent-5mkh5\" (UID: \"2882d280-bc58-4983-89f7-bfc3273fee8f\") " pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723721    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db6wd\" (UniqueName: \"kubernetes.io/projected/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-kube-api-access-db6wd\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723752    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovnkube-script-lib\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725121 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723837    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-tuned\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-host\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723895    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-netns\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723923    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-cni-multus\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723974    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da004608-b719-4ddd-b8fa-e55f4cb34081-cni-binary-copy\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.723993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-device-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724021    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-run-netns\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-socket-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-cnibin\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-registration-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724216    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-systemd-units\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovnkube-config\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovn-node-metrics-cert\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.725932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724343    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724370    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-serviceca\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724396    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-ovn\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724469    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-hostroot\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-daemon-config\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-run\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-hosts-file\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnlk\" (UniqueName: \"kubernetes.io/projected/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-kube-api-access-xgnlk\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724594    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-env-overrides\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724641    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724672    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-var-lib-kubelet\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a71ac77c-691c-4723-bc03-974cd9990fdb-tmp\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphh2\" (UniqueName: \"kubernetes.io/projected/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-kube-api-access-cphh2\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-etc-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724813    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovnkube-script-lib\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/da004608-b719-4ddd-b8fa-e55f4cb34081-kube-api-access-2zv7b\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.726706 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.724901    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.724919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-env-overrides\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.724977    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:18.224940118 +0000 UTC m=+2.984965743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725028    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-run-ovn\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-host-var-lib-cni-multus\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-etc-openvswitch\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725224    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725241    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7598ba-6577-4948-9df7-ba4528ee76b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-registration-dir\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725348    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/70c2aa93-4685-4fe5-a1c6-eecf73c43442-systemd-units\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725365    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725371    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-cni-dir\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725654    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a722293e-4534-4652-be28-8df3b206b801-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725797    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovnkube-config\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.725845    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da004608-b719-4ddd-b8fa-e55f4cb34081-hostroot\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.726219    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da004608-b719-4ddd-b8fa-e55f4cb34081-multus-daemon-config\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.727580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.726724    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.728481 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.726738    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df7598ba-6577-4948-9df7-ba4528ee76b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.729266 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.729230    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/70c2aa93-4685-4fe5-a1c6-eecf73c43442-ovn-node-metrics-cert\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.729401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.729382    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2882d280-bc58-4983-89f7-bfc3273fee8f-agent-certs\") pod \"konnectivity-agent-5mkh5\" (UID: \"2882d280-bc58-4983-89f7-bfc3273fee8f\") " pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.731820 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.731798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zmr\" (UniqueName: \"kubernetes.io/projected/70c2aa93-4685-4fe5-a1c6-eecf73c43442-kube-api-access-76zmr\") pod \"ovnkube-node-wl2tm\" (UID: \"70c2aa93-4685-4fe5-a1c6-eecf73c43442\") " pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.732817 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.732786    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68t2\" (UniqueName: \"kubernetes.io/projected/a722293e-4534-4652-be28-8df3b206b801-kube-api-access-w68t2\") pod \"aws-ebs-csi-driver-node-zb78d\" (UID: \"a722293e-4534-4652-be28-8df3b206b801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.733037 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.733015    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bx56\" (UniqueName: \"kubernetes.io/projected/0b662c0c-d267-40aa-8c94-748caf5b4735-kube-api-access-8bx56\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:17.735035 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.735010    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkmc\" (UniqueName: \"kubernetes.io/projected/df7598ba-6577-4948-9df7-ba4528ee76b9-kube-api-access-8rkmc\") pod \"multus-additional-cni-plugins-hvthd\" (UID: \"df7598ba-6577-4948-9df7-ba4528ee76b9\") " pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.735714 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.735686    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zv7b\" (UniqueName: \"kubernetes.io/projected/da004608-b719-4ddd-b8fa-e55f4cb34081-kube-api-access-2zv7b\") pod \"multus-4lvf4\" (UID: \"da004608-b719-4ddd-b8fa-e55f4cb34081\") " pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.825313 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysctl-d\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825313 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825322    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-host\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-kubelet-config\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-sys\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-systemd\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825426    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-iptables-alerter-script\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-dbus\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysctl-d\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-kubelet-config\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-sys\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825513    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-systemd\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825512    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-host\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-dbus\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:17.825544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysctl-conf\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.825579    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-tmp-dir\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825613    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysconfig\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.825634    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:18.325614621 +0000 UTC m=+3.085640239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825655    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysconfig\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825660    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-sysctl-conf\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825661    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-lib-modules\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825711    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-host-slash\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-modprobe-d\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825762    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-kubernetes\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825769    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-lib-modules\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825789    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztbk\" (UniqueName: \"kubernetes.io/projected/a71ac77c-691c-4723-bc03-974cd9990fdb-kube-api-access-dztbk\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db6wd\" (UniqueName: \"kubernetes.io/projected/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-kube-api-access-db6wd\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825833    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-kubernetes\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825838    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-modprobe-d\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825791    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-host-slash\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.826147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-tuned\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825900    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-host\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825938    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-tmp-dir\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-serviceca\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.825987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-run\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-host\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826010    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-hosts-file\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-hosts-file\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnlk\" (UniqueName: \"kubernetes.io/projected/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-kube-api-access-xgnlk\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-var-lib-kubelet\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826110    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a71ac77c-691c-4723-bc03-974cd9990fdb-tmp\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826128    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-run\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphh2\" (UniqueName: \"kubernetes.io/projected/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-kube-api-access-cphh2\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-iptables-alerter-script\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826176    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71ac77c-691c-4723-bc03-974cd9990fdb-var-lib-kubelet\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.826942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.826462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-serviceca\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.828293 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.828270    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a71ac77c-691c-4723-bc03-974cd9990fdb-tmp\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.828437 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.828413    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a71ac77c-691c-4723-bc03-974cd9990fdb-etc-tuned\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.837966 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.837910    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:17.837966 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.837935    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:17.837966 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.837949    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:17.838159 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:17.838009    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:18.337995828 +0000 UTC m=+3.098021437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:17.841531 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.841502    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db6wd\" (UniqueName: \"kubernetes.io/projected/1e62a1f6-bb7f-4671-abb0-021a0aed58cf-kube-api-access-db6wd\") pod \"iptables-alerter-96tjh\" (UID: \"1e62a1f6-bb7f-4671-abb0-021a0aed58cf\") " pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:17.842114 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.842095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztbk\" (UniqueName: \"kubernetes.io/projected/a71ac77c-691c-4723-bc03-974cd9990fdb-kube-api-access-dztbk\") pod \"tuned-w9m2f\" (UID: \"a71ac77c-691c-4723-bc03-974cd9990fdb\") " pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.842569 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.842548    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphh2\" (UniqueName: \"kubernetes.io/projected/3eb4e5d0-96cd-4008-a7a0-d75e4df4493e-kube-api-access-cphh2\") pod \"node-resolver-hll47\" (UID: \"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e\") " pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.842662 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.842623    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnlk\" (UniqueName: \"kubernetes.io/projected/6553e69a-e7c5-4d46-b1d1-3ba4a443b68b-kube-api-access-xgnlk\") pod \"node-ca-h5b8n\" (UID: \"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b\") " pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.911927 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.911903    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d"
Jul 28 17:10:17.928524 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.928503    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:17.937456 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.937437    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lvf4"
Jul 28 17:10:17.945219 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.945204    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hll47"
Jul 28 17:10:17.952963 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.952945    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvthd"
Jul 28 17:10:17.961820 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.961803    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:17.969486 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.969470    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9m2f"
Jul 28 17:10:17.977373 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.977356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h5b8n"
Jul 28 17:10:17.983134 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:17.983115    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-96tjh"
Jul 28 17:10:18.123882 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.123820    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:18.229199 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.229124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:18.229371 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.229264    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:18.229371 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.229346    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.229326118 +0000 UTC m=+3.989351732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:18.330260 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.330224    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:18.330416 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.330382    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:18.330479 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.330447    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.330430195 +0000 UTC m=+4.090455814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:18.431607 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.431539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:18.431768 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.431726    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:18.431768 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.431752    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:18.431768 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.431766    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:18.431930 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.431833    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.431813921 +0000 UTC m=+4.191839536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:18.666891 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.666855    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:17 +0000 UTC" deadline="2027-12-25 21:50:46.923620301 +0000 UTC"
Jul 28 17:10:18.666891 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.666886    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12364h40m28.256738242s"
Jul 28 17:10:18.702566 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:18.702503    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:18.702704 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:18.702622    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:18.799177 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.799148    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e62a1f6_bb7f_4671_abb0_021a0aed58cf.slice/crio-b1bd91adb3725f10e83b9e292aff4b216b56545c6d4226ed6a902f8d3a1d69db WatchSource:0}: Error finding container b1bd91adb3725f10e83b9e292aff4b216b56545c6d4226ed6a902f8d3a1d69db: Status 404 returned error can't find the container with id b1bd91adb3725f10e83b9e292aff4b216b56545c6d4226ed6a902f8d3a1d69db
Jul 28 17:10:18.800500 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.800472    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c2aa93_4685_4fe5_a1c6_eecf73c43442.slice/crio-ed9eff578dd427a686da0c9ea84952a7019bd9c7bd3627d774606230dcd28757 WatchSource:0}: Error finding container ed9eff578dd427a686da0c9ea84952a7019bd9c7bd3627d774606230dcd28757: Status 404 returned error can't find the container with id ed9eff578dd427a686da0c9ea84952a7019bd9c7bd3627d774606230dcd28757
Jul 28 17:10:18.802317 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.802235    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71ac77c_691c_4723_bc03_974cd9990fdb.slice/crio-9ef5d81953f4b90185458a8e5ff49695cfeab6301ddee1ba45f45574c220f2ab WatchSource:0}: Error finding container 9ef5d81953f4b90185458a8e5ff49695cfeab6301ddee1ba45f45574c220f2ab: Status 404 returned error can't find the container with id 9ef5d81953f4b90185458a8e5ff49695cfeab6301ddee1ba45f45574c220f2ab
Jul 28 17:10:18.804216 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.803724    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda722293e_4534_4652_be28_8df3b206b801.slice/crio-03cd1a7562e0a35a3ddc8af322a78c14e62ef83a0f5d66ccdf0a4a611131798b WatchSource:0}: Error finding container 03cd1a7562e0a35a3ddc8af322a78c14e62ef83a0f5d66ccdf0a4a611131798b: Status 404 returned error can't find the container with id 03cd1a7562e0a35a3ddc8af322a78c14e62ef83a0f5d66ccdf0a4a611131798b
Jul 28 17:10:18.804616 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.804514    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda004608_b719_4ddd_b8fa_e55f4cb34081.slice/crio-eaab81691fd370dea24a631dca61cb5a7091d2b1bae897005525e6a9d7b1cdfa WatchSource:0}: Error finding container eaab81691fd370dea24a631dca61cb5a7091d2b1bae897005525e6a9d7b1cdfa: Status 404 returned error can't find the container with id eaab81691fd370dea24a631dca61cb5a7091d2b1bae897005525e6a9d7b1cdfa
Jul 28 17:10:18.807748 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.807620    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7598ba_6577_4948_9df7_ba4528ee76b9.slice/crio-4e9ec40d739b10d340fa5ba86f3999a111595acf972f08387dfe97746c014eaf WatchSource:0}: Error finding container 4e9ec40d739b10d340fa5ba86f3999a111595acf972f08387dfe97746c014eaf: Status 404 returned error can't find the container with id 4e9ec40d739b10d340fa5ba86f3999a111595acf972f08387dfe97746c014eaf
Jul 28 17:10:18.809951 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.809907    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2882d280_bc58_4983_89f7_bfc3273fee8f.slice/crio-3216a8fee6529563f536115912b35c0f3bfbdffa6afafd58081229f125dd2ff9 WatchSource:0}: Error finding container 3216a8fee6529563f536115912b35c0f3bfbdffa6afafd58081229f125dd2ff9: Status 404 returned error can't find the container with id 3216a8fee6529563f536115912b35c0f3bfbdffa6afafd58081229f125dd2ff9
Jul 28 17:10:18.811636 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:18.811600    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6553e69a_e7c5_4d46_b1d1_3ba4a443b68b.slice/crio-d49d8be16bb4a05152a9a779043cca39ff4cb32f4327029a4bfb2f07d006b09f WatchSource:0}: Error finding container d49d8be16bb4a05152a9a779043cca39ff4cb32f4327029a4bfb2f07d006b09f: Status 404 returned error can't find the container with id d49d8be16bb4a05152a9a779043cca39ff4cb32f4327029a4bfb2f07d006b09f
Jul 28 17:10:19.237157 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.237085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:19.237329 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.237275    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:19.237411 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.237343    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.237321889 +0000 UTC m=+5.997347497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:19.338383 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.338346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:19.338549 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.338530    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:19.338617 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.338604    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.338583241 +0000 UTC m=+6.098608872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:19.440084 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.439450    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:19.440084 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.439652    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:19.440084 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.439672    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:19.440084 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.439685    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:19.440084 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.439740    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.439722454 +0000 UTC m=+6.199748078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:19.669606 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.669472    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:17 +0000 UTC" deadline="2028-04-24 11:33:23.157644242 +0000 UTC"
Jul 28 17:10:19.669606 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.669532    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15258h23m3.488116218s"
Jul 28 17:10:19.704001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.703287    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:19.704001 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.703437    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:19.704001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.703855    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:19.704001 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:19.703956    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:19.723392 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.723340    2571 generic.go:356] "Generic (PLEG): container finished" podID="2b5e167bbe3082ae5f67dd557ac7ea1b" containerID="96d13aeb61677066ba9359047b4ce76f949059c86cc5acf32e875c3a9ed537e6" exitCode=0
Jul 28 17:10:19.723625 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.723441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" event={"ID":"2b5e167bbe3082ae5f67dd557ac7ea1b","Type":"ContainerDied","Data":"96d13aeb61677066ba9359047b4ce76f949059c86cc5acf32e875c3a9ed537e6"}
Jul 28 17:10:19.748877 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.748313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal" event={"ID":"a1804774383ef499fd25a45e880ee305","Type":"ContainerStarted","Data":"c73c3c1b12013a8a0e89208c1995c76079ce689fea6fb9cf943d91a26563761f"}
Jul 28 17:10:19.760725 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.760648    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5b8n" event={"ID":"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b","Type":"ContainerStarted","Data":"d49d8be16bb4a05152a9a779043cca39ff4cb32f4327029a4bfb2f07d006b09f"}
Jul 28 17:10:19.763448 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.763395    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5mkh5" event={"ID":"2882d280-bc58-4983-89f7-bfc3273fee8f","Type":"ContainerStarted","Data":"3216a8fee6529563f536115912b35c0f3bfbdffa6afafd58081229f125dd2ff9"}
Jul 28 17:10:19.781419 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.781368    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerStarted","Data":"4e9ec40d739b10d340fa5ba86f3999a111595acf972f08387dfe97746c014eaf"}
Jul 28 17:10:19.802744 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.802712    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lvf4" event={"ID":"da004608-b719-4ddd-b8fa-e55f4cb34081","Type":"ContainerStarted","Data":"eaab81691fd370dea24a631dca61cb5a7091d2b1bae897005525e6a9d7b1cdfa"}
Jul 28 17:10:19.813468 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.813441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-96tjh" event={"ID":"1e62a1f6-bb7f-4671-abb0-021a0aed58cf","Type":"ContainerStarted","Data":"b1bd91adb3725f10e83b9e292aff4b216b56545c6d4226ed6a902f8d3a1d69db"}
Jul 28 17:10:19.830673 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.830615    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hll47" event={"ID":"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e","Type":"ContainerStarted","Data":"f8dbf27998ed698756975cd435dbfc10627647a1688b4401ab9fa284e745a4b0"}
Jul 28 17:10:19.851379 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.851282    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"ed9eff578dd427a686da0c9ea84952a7019bd9c7bd3627d774606230dcd28757"}
Jul 28 17:10:19.863439 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.863414    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d" event={"ID":"a722293e-4534-4652-be28-8df3b206b801","Type":"ContainerStarted","Data":"03cd1a7562e0a35a3ddc8af322a78c14e62ef83a0f5d66ccdf0a4a611131798b"}
Jul 28 17:10:19.882100 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:19.882050    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9m2f" event={"ID":"a71ac77c-691c-4723-bc03-974cd9990fdb","Type":"ContainerStarted","Data":"9ef5d81953f4b90185458a8e5ff49695cfeab6301ddee1ba45f45574c220f2ab"}
Jul 28 17:10:20.702792 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:20.702310    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:20.702792 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:20.702433    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:20.892706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:20.892640    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" event={"ID":"2b5e167bbe3082ae5f67dd557ac7ea1b","Type":"ContainerStarted","Data":"ff0d6465b81cdae6cf3be8393da5ce38ac4ddc3992a89f00212c2be1443422ad"}
Jul 28 17:10:20.912379 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:20.912099    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-61.ec2.internal" podStartSLOduration=3.912079144 podStartE2EDuration="3.912079144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:19.766980797 +0000 UTC m=+4.527006430" watchObservedRunningTime="2026-07-28 17:10:20.912079144 +0000 UTC m=+5.672104774"
Jul 28 17:10:21.252120 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:21.252039    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:21.252292 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.252180    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.252292 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.252235    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:25.252217156 +0000 UTC m=+10.012242766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.352966 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:21.352935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:21.353138 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.353071    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.353138 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.353126    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:25.353109837 +0000 UTC m=+10.113135468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.454176 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:21.454141    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:21.454375 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.454345    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:21.454375 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.454372    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:21.454494 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.454385    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:21.454494 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.454461    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:25.454421724 +0000 UTC m=+10.214447333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:21.703290 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:21.703200    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:21.703290 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:21.703274    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:21.703750 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.703389    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:21.703817 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:21.703796    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:22.703054 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:22.702937    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:22.703233 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:22.703065    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:23.703136 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:23.702488    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:23.703136 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:23.702619    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:23.703136 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:23.702999    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:23.703136 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:23.703092    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:24.702356 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:24.702274    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:24.702521 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:24.702385    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:25.286345 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:25.286306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:25.286807 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.286464    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:25.286807 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.286524    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:33.286505938 +0000 UTC m=+18.046531550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:25.387453 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:25.387417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:25.387609 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.387582    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:25.387657 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.387646    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:33.387632207 +0000 UTC m=+18.147657813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:25.487823 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:25.487793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:25.487985 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.487918    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:25.487985 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.487939    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:25.487985 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.487953    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:25.488153 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.488018    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:33.487992563 +0000 UTC m=+18.248018189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:25.704310 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:25.704089    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:25.704310 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.704220    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:25.704627 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:25.704597    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:25.704715 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:25.704676    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:26.703010 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:26.702825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:26.703010 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:26.702942    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:27.703104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:27.702992    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:27.703104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:27.703037    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:27.703665 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:27.703127    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:27.703665 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:27.703199    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:28.702894 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:28.702864    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:28.703066 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:28.702981    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:29.703222 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:29.703141    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:29.703643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:29.703272    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:29.703643 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:29.703150    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:29.703643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:29.703428    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:30.703261 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:30.703219    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:30.703695 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:30.703351    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:31.702475 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:31.702413    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:31.702647 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:31.702555    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:31.702647 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:31.702615    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:31.702745 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:31.702728    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:32.703124 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:32.703088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:32.703645 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:32.703209    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:33.356600 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:33.356567    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:33.356765 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.356691    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:33.356879 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.356763    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:49.356743837 +0000 UTC m=+34.116769447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:33.457221 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:33.457184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:33.457384 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.457344    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:33.457452 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.457407    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:49.457394228 +0000 UTC m=+34.217419850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:33.558393 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:33.558360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:33.558566 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.558551    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:33.558612 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.558580    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:33.558612 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.558594    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:33.558677 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.558656    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:49.558638531 +0000 UTC m=+34.318664151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:33.703178 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:33.703101    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:33.703596 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:33.703109    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:33.703596 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.703227    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:33.703596 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:33.703325    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:34.703068 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:34.703038    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:34.703240 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:34.703154    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:35.702873 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:35.702840    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:35.703046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:35.702928    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:35.703046 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:35.702946    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:35.703167 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:35.703038    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:36.702460 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:36.702424    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:36.702911 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:36.702539    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:37.703423 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.703169    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:37.703976 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.703214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:37.703976 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:37.703527    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:37.703976 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:37.703550    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:37.925548 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.925498    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h5b8n" event={"ID":"6553e69a-e7c5-4d46-b1d1-3ba4a443b68b","Type":"ContainerStarted","Data":"79880567f89bbafa24bbe2b4196f8acbedbaf9701aa8578369e80fda09ed568c"}
Jul 28 17:10:37.926842 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.926818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5mkh5" event={"ID":"2882d280-bc58-4983-89f7-bfc3273fee8f","Type":"ContainerStarted","Data":"91694111b186be1e760ff6e3d2a85b1e8d7f848a9473a80338d88a0d01ce59df"}
Jul 28 17:10:37.928170 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.928148    2571 generic.go:356] "Generic (PLEG): container finished" podID="df7598ba-6577-4948-9df7-ba4528ee76b9" containerID="b744c4b097fe0a2347803ab4b374d8dda663d6246c2d179ba74684be704cb6cb" exitCode=0
Jul 28 17:10:37.928240 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.928218    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerDied","Data":"b744c4b097fe0a2347803ab4b374d8dda663d6246c2d179ba74684be704cb6cb"}
Jul 28 17:10:37.929876 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.929845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lvf4" event={"ID":"da004608-b719-4ddd-b8fa-e55f4cb34081","Type":"ContainerStarted","Data":"fc152167e1494b5bfbd7335a607b00373f5eeb12ec31af3fc0d826b4d2940518"}
Jul 28 17:10:37.931334 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.931312    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hll47" event={"ID":"3eb4e5d0-96cd-4008-a7a0-d75e4df4493e","Type":"ContainerStarted","Data":"714fa37d0a6fcb993a9f915dbe369ebcdb22ff727bc61a31ff8524159e9b0c46"}
Jul 28 17:10:37.934044 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.934025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"982a125ab845b061765fdcccae0476071a01c0feb94a32f66db9922892e56a6c"}
Jul 28 17:10:37.934138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.934074    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"e6871b9ecc5dc9ba0b4ec0eed579d28c189f88f1099d30f886ab9c43627ac59c"}
Jul 28 17:10:37.934138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.934130    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"3204bcd8a1738794a5e532c9e087729f00687b6c248c43d446dc9e31523df137"}
Jul 28 17:10:37.934236 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.934168    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"c73b97f78b3123048b54f2b52a7f7d90d84ead37745ef1daf8409ffa8f9ea5e9"}
Jul 28 17:10:37.934236 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.934211    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"1400dfc0d1d7af32bc9d2b57155efa601f36b256714c0c95f610ec6790eec946"}
Jul 28 17:10:37.934352 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.934278    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"34ca5043858f31215dcc9e4b0543437a53833c29c83d016d3f214cab191ca634"}
Jul 28 17:10:37.935340 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.935322    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d" event={"ID":"a722293e-4534-4652-be28-8df3b206b801","Type":"ContainerStarted","Data":"4e788cf742c717455b1177a820c78c585e0c1deb4342f783e3356fdcffc6f2ec"}
Jul 28 17:10:37.936472 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.936453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9m2f" event={"ID":"a71ac77c-691c-4723-bc03-974cd9990fdb","Type":"ContainerStarted","Data":"0111d895f4dead683dc15b029d6ee0e180e74fec12dbd66daa26378f6b69db9b"}
Jul 28 17:10:37.940970 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.940935    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-61.ec2.internal" podStartSLOduration=20.940903845 podStartE2EDuration="20.940903845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:20.9123527 +0000 UTC m=+5.672378329" watchObservedRunningTime="2026-07-28 17:10:37.940903845 +0000 UTC m=+22.700929452"
Jul 28 17:10:37.941327 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.941306    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h5b8n" podStartSLOduration=3.63335576 podStartE2EDuration="21.941297666s" totalImagesPullingTime="18.307941906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:37.940862545 +0000 UTC m=+22.700888172" watchObservedRunningTime="2026-07-28 17:10:37.941297666 +0000 UTC m=+22.701323293"
Jul 28 17:10:37.957588 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.957557    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5mkh5" podStartSLOduration=7.821127886 podStartE2EDuration="21.957544913s" totalImagesPullingTime="14.136417027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:37.957162762 +0000 UTC m=+22.717188388" watchObservedRunningTime="2026-07-28 17:10:37.957544913 +0000 UTC m=+22.717570540"
Jul 28 17:10:37.985398 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:37.985364    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hll47" podStartSLOduration=3.630827433 podStartE2EDuration="21.985352201s" totalImagesPullingTime="18.354524768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:37.985192913 +0000 UTC m=+22.745218540" watchObservedRunningTime="2026-07-28 17:10:37.985352201 +0000 UTC m=+22.745377828"
Jul 28 17:10:38.006180 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.006148    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4lvf4" podStartSLOduration=3.505743189 podStartE2EDuration="22.006137498s" totalImagesPullingTime="18.500394309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.006082015 +0000 UTC m=+22.766107653" watchObservedRunningTime="2026-07-28 17:10:38.006137498 +0000 UTC m=+22.766163126"
Jul 28 17:10:38.047644 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.047608    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w9m2f" podStartSLOduration=3.683589141 podStartE2EDuration="22.047596719s" totalImagesPullingTime="18.364007578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.047477045 +0000 UTC m=+22.807502673" watchObservedRunningTime="2026-07-28 17:10:38.047596719 +0000 UTC m=+22.807622346"
Jul 28 17:10:38.670321 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.670295    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:38.670940 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.670920    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:38.702392 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.702371    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:38.702505 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:38.702453    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:38.737008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.736978    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:10:38.939657 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.939588    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-96tjh" event={"ID":"1e62a1f6-bb7f-4671-abb0-021a0aed58cf","Type":"ContainerStarted","Data":"f5f79ee8f2ceb5ea501f0a27d4bc551c0db3072f1ec141fd915d71e0cda41edc"}
Jul 28 17:10:38.941256 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.941219    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d" event={"ID":"a722293e-4534-4652-be28-8df3b206b801","Type":"ContainerStarted","Data":"be681e04ab97a7ba017bad536c79d1424706b78c244e55ab6e4e51f239dd5482"}
Jul 28 17:10:38.941979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.941932    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:38.942291 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.942275    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5mkh5"
Jul 28 17:10:38.956316 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:38.956282    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-96tjh" podStartSLOduration=4.634717645 podStartE2EDuration="22.95626936s" totalImagesPullingTime="18.321551715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:38.955900704 +0000 UTC m=+23.715926331" watchObservedRunningTime="2026-07-28 17:10:38.95626936 +0000 UTC m=+23.716294996"
Jul 28 17:10:39.663578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:39.663467    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:10:38.737006567Z","UUID":"672ad338-6675-45f7-8cb9-d8b01d25b847","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:10:39.665046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:39.665028    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:10:39.665162 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:39.665063    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:10:39.702393 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:39.702359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:39.702556 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:39.702402    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:39.702556 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:39.702493    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:39.702679 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:39.702586    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:40.702448 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:40.702223    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:40.702916 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:40.702483    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:40.949880 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:40.949845    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"159af1d99408f21add0a431dcbba3026ce4221710289c201d206f63db89dcc6b"}
Jul 28 17:10:40.952069 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:40.952036    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d" event={"ID":"a722293e-4534-4652-be28-8df3b206b801","Type":"ContainerStarted","Data":"082867ebefed2522d2bd21c03239df78d158941055189176b401b77be98ddc3c"}
Jul 28 17:10:40.973184 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:40.973109    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb78d" podStartSLOduration=3.899712997 podStartE2EDuration="24.973095455s" totalImagesPullingTime="21.073382458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:40.972750264 +0000 UTC m=+25.732775894" watchObservedRunningTime="2026-07-28 17:10:40.973095455 +0000 UTC m=+25.733121083"
Jul 28 17:10:41.702438 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:41.702409    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:41.702609 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:41.702539    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:41.702609 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:41.702599    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:41.703127 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:41.702699    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:42.702865 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.702717    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:42.703375 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:42.702926    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:42.959628 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.959557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" event={"ID":"70c2aa93-4685-4fe5-a1c6-eecf73c43442","Type":"ContainerStarted","Data":"96a2b37cfde9581d652f58befdeca73cf60907aabd6c8abf6e5b76e141c218af"}
Jul 28 17:10:42.959830 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.959814    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:42.961432 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.961406    2571 generic.go:356] "Generic (PLEG): container finished" podID="df7598ba-6577-4948-9df7-ba4528ee76b9" containerID="ddbea23b97871a6136931fb9c040b1307c469703971d053db1776eb39238a8af" exitCode=0
Jul 28 17:10:42.961525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.961440    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerDied","Data":"ddbea23b97871a6136931fb9c040b1307c469703971d053db1776eb39238a8af"}
Jul 28 17:10:42.975109 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.975091    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:42.997503 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:42.997469    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm" podStartSLOduration=8.585772176999999 podStartE2EDuration="26.997457128s" totalImagesPullingTime="18.411684951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.996617264 +0000 UTC m=+27.756642891" watchObservedRunningTime="2026-07-28 17:10:42.997457128 +0000 UTC m=+27.757482755"
Jul 28 17:10:43.702562 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:43.702533    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:43.702778 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:43.702667    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:43.702778 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:43.702752    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:43.702893 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:43.702867    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:43.963214 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:43.963195    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:10:43.963612 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:43.963595    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:43.978067 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:43.978048    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:44.089089 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.089063    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rqtb"]
Jul 28 17:10:44.089184 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.089163    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:44.089278 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:44.089237    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:44.095005 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.094986    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z4x8n"]
Jul 28 17:10:44.095080 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.095070    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:44.095155 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:44.095141    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:44.099726 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.099703    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-22bpl"]
Jul 28 17:10:44.099805 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.099787    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:44.099901 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:44.099877    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:44.966918 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.966885    2571 generic.go:356] "Generic (PLEG): container finished" podID="df7598ba-6577-4948-9df7-ba4528ee76b9" containerID="7b5a954d825bb72a75bdff6406d85bbf8ac0b3fc91b7289670d00a162a24c174" exitCode=0
Jul 28 17:10:44.967366 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.966960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerDied","Data":"7b5a954d825bb72a75bdff6406d85bbf8ac0b3fc91b7289670d00a162a24c174"}
Jul 28 17:10:44.967366 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:44.967016    2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:10:45.116030 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:45.116005    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:10:45.704665 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:45.704546    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:45.704841 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:45.704703    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:45.706061 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:45.705117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:45.706061 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:45.705238    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:45.706061 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:45.705312    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:45.706061 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:45.705416    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:46.975509 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:46.975292    2571 generic.go:356] "Generic (PLEG): container finished" podID="df7598ba-6577-4948-9df7-ba4528ee76b9" containerID="0ee6600464c6305c51eee4955fe9d4c4ca3849ce9fb2f5b42fe98fa600f4222a" exitCode=0
Jul 28 17:10:46.975509 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:46.975364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerDied","Data":"0ee6600464c6305c51eee4955fe9d4c4ca3849ce9fb2f5b42fe98fa600f4222a"}
Jul 28 17:10:47.702649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:47.702627    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:47.702649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:47.702640    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:47.702796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:47.702640    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:47.702796 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:47.702717    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:47.702861 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:47.702804    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:47.702894 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:47.702867    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:49.384990 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:49.384957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:49.385518 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.385071    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:49.385518 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.385123    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:21.385110676 +0000 UTC m=+66.145136283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:49.485621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:49.485590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:49.485766 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.485695    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:49.485766 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.485751    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret podName:e5b4bddb-be20-4289-bc3b-0b64b94a71b7 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:21.485734103 +0000 UTC m=+66.245759710 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret") pod "global-pull-secret-syncer-5rqtb" (UID: "e5b4bddb-be20-4289-bc3b-0b64b94a71b7") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:49.586204 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:49.586171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:49.586358 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.586305    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:49.586358 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.586327    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:49.586358 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.586336    2571 projected.go:196] Error preparing data for projected volume kube-api-access-x9vk7 for pod openshift-network-diagnostics/network-check-target-22bpl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:49.586474 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.586384    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7 podName:41dbd98a-cb15-4862-8ff6-b516258b3076 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:21.586368562 +0000 UTC m=+66.346394185 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9vk7" (UniqueName: "kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7") pod "network-check-target-22bpl" (UID: "41dbd98a-cb15-4862-8ff6-b516258b3076") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:49.703241 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:49.703214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:49.703391 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.703368    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z4x8n" podUID="0b662c0c-d267-40aa-8c94-748caf5b4735"
Jul 28 17:10:49.703482 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:49.703464    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:49.703575 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.703553    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rqtb" podUID="e5b4bddb-be20-4289-bc3b-0b64b94a71b7"
Jul 28 17:10:49.703641 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:49.703615    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:49.703735 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:49.703717    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-22bpl" podUID="41dbd98a-cb15-4862-8ff6-b516258b3076"
Jul 28 17:10:50.095509 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.095433    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-61.ec2.internal" event="NodeReady"
Jul 28 17:10:50.095667 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.095577    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:10:50.147153 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.147121    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7458574f8f-q7bxs"]
Jul 28 17:10:50.149581 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.149561    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.152193 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.152165    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:10:50.152300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.152200    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:10:50.152710 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.152690    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-chqkr\""
Jul 28 17:10:50.152827 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.152759    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:10:50.158523 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.158504    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:10:50.163059 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.163036    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6k8qm"]
Jul 28 17:10:50.165182 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.165165    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jc2sx"]
Jul 28 17:10:50.165327 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.165310    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.167178 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.167159    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7458574f8f-q7bxs"]
Jul 28 17:10:50.167290 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.167261    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.168061 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.168041    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:10:50.168171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.168070    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-947wh\""
Jul 28 17:10:50.168171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.168048    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:10:50.169686 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.169667    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:10:50.170076 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.170018    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:10:50.175589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.170266    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:10:50.175589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.170834    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sx2rm\""
Jul 28 17:10:50.175589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.173594    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jc2sx"]
Jul 28 17:10:50.175589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.175169    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6k8qm"]
Jul 28 17:10:50.292122 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292089    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-certificates\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm22\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-kube-api-access-nhm22\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1a87c97-7a97-4e12-8caa-7f2d96d61ffc-cert\") pod \"ingress-canary-jc2sx\" (UID: \"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc\") " pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.292301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292180    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8wx\" (UniqueName: \"kubernetes.io/projected/a5a94646-a349-4508-8f0d-b7511f8c1c44-kube-api-access-2h8wx\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.292301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-bound-sa-token\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a94646-a349-4508-8f0d-b7511f8c1c44-config-volume\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.292301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292295    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.292534 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-image-registry-private-configuration\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292534 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292342    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5fd7f1f-0140-468c-88e5-c96d27d2a140-ca-trust-extracted\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292534 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-installation-pull-secrets\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292534 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.292534 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrbp\" (UniqueName: \"kubernetes.io/projected/b1a87c97-7a97-4e12-8caa-7f2d96d61ffc-kube-api-access-4rrbp\") pod \"ingress-canary-jc2sx\" (UID: \"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc\") " pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.292721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292565    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a94646-a349-4508-8f0d-b7511f8c1c44-tmp-dir\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.292721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.292609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-trusted-ca\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.393913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.393886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-image-registry-private-configuration\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.393919    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5fd7f1f-0140-468c-88e5-c96d27d2a140-ca-trust-extracted\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.393935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-installation-pull-secrets\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.393966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.393989    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrbp\" (UniqueName: \"kubernetes.io/projected/b1a87c97-7a97-4e12-8caa-7f2d96d61ffc-kube-api-access-4rrbp\") pod \"ingress-canary-jc2sx\" (UID: \"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc\") " pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394025    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a94646-a349-4508-8f0d-b7511f8c1c44-tmp-dir\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394053    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-trusted-ca\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-certificates\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm22\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-kube-api-access-nhm22\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394125    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1a87c97-7a97-4e12-8caa-7f2d96d61ffc-cert\") pod \"ingress-canary-jc2sx\" (UID: \"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc\") " pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394145    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8wx\" (UniqueName: \"kubernetes.io/projected/a5a94646-a349-4508-8f0d-b7511f8c1c44-kube-api-access-2h8wx\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-bound-sa-token\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a94646-a349-4508-8f0d-b7511f8c1c44-config-volume\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.394325 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394230    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394334    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5fd7f1f-0140-468c-88e5-c96d27d2a140-ca-trust-extracted\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.394345    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.394412    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:50.894393368 +0000 UTC m=+35.654418974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.394424    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.394441    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.394502    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:50.89448433 +0000 UTC m=+35.654509939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:10:50.395000 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.394912    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a94646-a349-4508-8f0d-b7511f8c1c44-tmp-dir\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.395384 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.395210    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-certificates\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.395384 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.395281    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a94646-a349-4508-8f0d-b7511f8c1c44-config-volume\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.395481 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.395436    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-trusted-ca\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.399094 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.399069    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-image-registry-private-configuration\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.399094 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.399082    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-installation-pull-secrets\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.399236 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.399126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1a87c97-7a97-4e12-8caa-7f2d96d61ffc-cert\") pod \"ingress-canary-jc2sx\" (UID: \"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc\") " pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.403454 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.403430    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-bound-sa-token\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.403563 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.403470    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm22\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-kube-api-access-nhm22\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.403822 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.403803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8wx\" (UniqueName: \"kubernetes.io/projected/a5a94646-a349-4508-8f0d-b7511f8c1c44-kube-api-access-2h8wx\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.403958 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.403937    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrbp\" (UniqueName: \"kubernetes.io/projected/b1a87c97-7a97-4e12-8caa-7f2d96d61ffc-kube-api-access-4rrbp\") pod \"ingress-canary-jc2sx\" (UID: \"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc\") " pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.484624 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.484594    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jc2sx"
Jul 28 17:10:50.898948 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.898910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:50.899117 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:50.898984    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:50.899117 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.899079    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:50.899221 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.899140    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:50.899221 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.899152    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.899132847 +0000 UTC m=+36.659158454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:50.899221 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.899159    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:10:50.899221 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:50.899211    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.899195151 +0000 UTC m=+36.659220779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:10:51.353786 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.353597    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jc2sx"]
Jul 28 17:10:51.357626 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:10:51.357583    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a87c97_7a97_4e12_8caa_7f2d96d61ffc.slice/crio-5d4c802db982b34ec0e3ddb11def3ac4be328a6c47e3df7981b1ab31ae916641 WatchSource:0}: Error finding container 5d4c802db982b34ec0e3ddb11def3ac4be328a6c47e3df7981b1ab31ae916641: Status 404 returned error can't find the container with id 5d4c802db982b34ec0e3ddb11def3ac4be328a6c47e3df7981b1ab31ae916641
Jul 28 17:10:51.702821 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.702795    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:10:51.703333 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.702794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:10:51.703333 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.702794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:10:51.706692 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.706676    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:10:51.706786 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.706693    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxs5k\""
Jul 28 17:10:51.707912 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.707895    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:10:51.707991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.707972    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sw5vg\""
Jul 28 17:10:51.708108 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.708094    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:10:51.708169 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.708153    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:10:51.906938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.906910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:51.907093 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.907000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:51.907093 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:51.907078    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:51.907195 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:51.907097    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:10:51.907195 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:51.907114    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:51.907195 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:51.907169    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:53.907148996 +0000 UTC m=+38.667174603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:10:51.907195 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:51.907186    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:53.907177319 +0000 UTC m=+38.667202931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:51.989593 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.989515    2571 generic.go:356] "Generic (PLEG): container finished" podID="df7598ba-6577-4948-9df7-ba4528ee76b9" containerID="94acca49d95c65212cf394dad2904e0e90147c83da21358c9b81d106dd67a6b2" exitCode=0
Jul 28 17:10:51.989724 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.989609    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerDied","Data":"94acca49d95c65212cf394dad2904e0e90147c83da21358c9b81d106dd67a6b2"}
Jul 28 17:10:51.990919 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:51.990888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jc2sx" event={"ID":"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc","Type":"ContainerStarted","Data":"5d4c802db982b34ec0e3ddb11def3ac4be328a6c47e3df7981b1ab31ae916641"}
Jul 28 17:10:52.997460 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:52.997413    2571 generic.go:356] "Generic (PLEG): container finished" podID="df7598ba-6577-4948-9df7-ba4528ee76b9" containerID="795107c13d77f2bcb0ad2f6d8910015f0c24e288cc7795014b266f522ac2a5a3" exitCode=0
Jul 28 17:10:52.997831 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:52.997456    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerDied","Data":"795107c13d77f2bcb0ad2f6d8910015f0c24e288cc7795014b266f522ac2a5a3"}
Jul 28 17:10:53.923938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:53.923753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:53.923938 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:53.923894    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:53.923938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:53.923930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:53.924101 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:53.923947    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:57.923931878 +0000 UTC m=+42.683957694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:53.924101 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:53.923999    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:53.924101 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:53.924010    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:10:53.924101 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:53.924046    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:57.924035858 +0000 UTC m=+42.684061464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:10:54.001173 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:54.001138    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jc2sx" event={"ID":"b1a87c97-7a97-4e12-8caa-7f2d96d61ffc","Type":"ContainerStarted","Data":"9638a47ee6aebb87fa63e05e1afa4841be3aa8305d2aae4ee6e3246f0f3462d8"}
Jul 28 17:10:54.004555 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:54.004526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvthd" event={"ID":"df7598ba-6577-4948-9df7-ba4528ee76b9","Type":"ContainerStarted","Data":"2c562f08b230e4d7c80b151c1292887c1d1d62288b61bba0f25949d8f0c18360"}
Jul 28 17:10:54.020488 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:54.020438    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jc2sx" podStartSLOduration=1.605652367 podStartE2EDuration="4.020418163s" totalImagesPullingTime="2.414765796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:54.019619087 +0000 UTC m=+38.779644718" watchObservedRunningTime="2026-07-28 17:10:54.020418163 +0000 UTC m=+38.780443792"
Jul 28 17:10:54.051480 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:54.051436    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hvthd" podStartSLOduration=9.253338318 podStartE2EDuration="38.051421105s" totalImagesPullingTime="28.798082787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:54.049752262 +0000 UTC m=+38.809777889" watchObservedRunningTime="2026-07-28 17:10:54.051421105 +0000 UTC m=+38.811446733"
Jul 28 17:10:54.264610 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:54.264547    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:54.390299 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:54.390272    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:57.950220 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:57.950191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:10:57.950643 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:10:57.950234    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:10:57.950643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:57.950333    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:57.950643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:57.950344    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:10:57.950643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:57.950383    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:05.950371725 +0000 UTC m=+50.710397331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:10:57.950643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:57.950334    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:57.950643 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:10:57.950454    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:05.950442182 +0000 UTC m=+50.710467789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:00.367987 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:00.367956    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:00.426729 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:00.426700    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:04.300328 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:04.300281    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:04.400853 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:04.400823    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:04.460923 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:04.460901    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:06.003811 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:06.003789    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:11:06.004135 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:06.003842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:11:06.004135 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:06.003917    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:06.004135 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:06.003920    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:06.004135 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:06.003935    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:11:06.004135 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:06.003971    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:22.003958043 +0000 UTC m=+66.763983649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:06.004135 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:06.003983    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:22.003977824 +0000 UTC m=+66.764003430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:11:10.450993 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.450966    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"]
Jul 28 17:11:10.452775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.452760    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"]
Jul 28 17:11:10.452932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.452909    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.455804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.455020    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.455804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.455606    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:11:10.455804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.455796    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:11:10.456054 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.455832    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 17:11:10.456763 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.456733    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:11:10.457382 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.457358    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 17:11:10.458772 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.458756    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rnfqq\""
Jul 28 17:11:10.466499 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.466482    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"]
Jul 28 17:11:10.467120 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.467103    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"]
Jul 28 17:11:10.504065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.504044    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"]
Jul 28 17:11:10.505776 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.505762    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.508389 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.508371    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:11:10.508581 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.508555    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:11:10.508741 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.508726    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:11:10.508812 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.508759    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:11:10.523117 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.523098    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"]
Jul 28 17:11:10.532771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.532749    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrw7\" (UniqueName: \"kubernetes.io/projected/f0e7ffe4-6286-4ae4-8ad3-81513b136398-kube-api-access-ffrw7\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.532863 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.532785    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ca82ebd1-86d8-450e-bd95-0c50cebff7e0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68c5675bf8-qq5c4\" (UID: \"ca82ebd1-86d8-450e-bd95-0c50cebff7e0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.532970 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.532821    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-ca\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.533179 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.533264 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-tmp\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.533327 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533276    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8rz\" (UniqueName: \"kubernetes.io/projected/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-kube-api-access-sq8rz\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.533452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-hub\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.533503 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7ffe4-6286-4ae4-8ad3-81513b136398-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.533561 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.533611 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.533680 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.533665    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r42\" (UniqueName: \"kubernetes.io/projected/ca82ebd1-86d8-450e-bd95-0c50cebff7e0-kube-api-access-l8r42\") pod \"managed-serviceaccount-addon-agent-68c5675bf8-qq5c4\" (UID: \"ca82ebd1-86d8-450e-bd95-0c50cebff7e0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.634557 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634535    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r42\" (UniqueName: \"kubernetes.io/projected/ca82ebd1-86d8-450e-bd95-0c50cebff7e0-kube-api-access-l8r42\") pod \"managed-serviceaccount-addon-agent-68c5675bf8-qq5c4\" (UID: \"ca82ebd1-86d8-450e-bd95-0c50cebff7e0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.634650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634561    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrw7\" (UniqueName: \"kubernetes.io/projected/f0e7ffe4-6286-4ae4-8ad3-81513b136398-kube-api-access-ffrw7\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.634650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ca82ebd1-86d8-450e-bd95-0c50cebff7e0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68c5675bf8-qq5c4\" (UID: \"ca82ebd1-86d8-450e-bd95-0c50cebff7e0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.634650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-ca\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.634650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634613    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.634650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-tmp\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.634892 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8rz\" (UniqueName: \"kubernetes.io/projected/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-kube-api-access-sq8rz\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.634892 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634744    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-hub\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.634892 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7ffe4-6286-4ae4-8ad3-81513b136398-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.634892 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634824    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.634892 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.634861    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.635171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.635148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-tmp\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.635497 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.635467    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e7ffe4-6286-4ae4-8ad3-81513b136398-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.638603 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.638582    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.638681 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.638583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-ca\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.638720 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.638708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.638788 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.638767    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f0e7ffe4-6286-4ae4-8ad3-81513b136398-hub\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.638929 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.638915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ca82ebd1-86d8-450e-bd95-0c50cebff7e0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-68c5675bf8-qq5c4\" (UID: \"ca82ebd1-86d8-450e-bd95-0c50cebff7e0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.638971 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.638951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.643455 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.643428    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrw7\" (UniqueName: \"kubernetes.io/projected/f0e7ffe4-6286-4ae4-8ad3-81513b136398-kube-api-access-ffrw7\") pod \"cluster-proxy-proxy-agent-6f6798d56b-rjfvp\" (UID: \"f0e7ffe4-6286-4ae4-8ad3-81513b136398\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.643554 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.643536    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8rz\" (UniqueName: \"kubernetes.io/projected/c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0-kube-api-access-sq8rz\") pod \"klusterlet-addon-workmgr-5c95c6f98b-rwmxb\" (UID: \"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.643886 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.643865    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r42\" (UniqueName: \"kubernetes.io/projected/ca82ebd1-86d8-450e-bd95-0c50cebff7e0-kube-api-access-l8r42\") pod \"managed-serviceaccount-addon-agent-68c5675bf8-qq5c4\" (UID: \"ca82ebd1-86d8-450e-bd95-0c50cebff7e0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.764843 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.764793    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:10.783040 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.783017    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"
Jul 28 17:11:10.813426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.813395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:11:10.922791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.922762    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"]
Jul 28 17:11:10.925714 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:11:10.925683    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bbf2f5_cda2_41bb_96e8_aeb3dcf209b0.slice/crio-c5cddb72f70143b2b2e65ca1a85af667929f9c415a8b1d56ab4be7c2ac784f6e WatchSource:0}: Error finding container c5cddb72f70143b2b2e65ca1a85af667929f9c415a8b1d56ab4be7c2ac784f6e: Status 404 returned error can't find the container with id c5cddb72f70143b2b2e65ca1a85af667929f9c415a8b1d56ab4be7c2ac784f6e
Jul 28 17:11:10.940699 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.940672    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4"]
Jul 28 17:11:10.943933 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:11:10.943913    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca82ebd1_86d8_450e_bd95_0c50cebff7e0.slice/crio-2fa0cb2b0b5cbf5d95831a1b040aa2f2cb57041856ae8a6651dafbcefce9f912 WatchSource:0}: Error finding container 2fa0cb2b0b5cbf5d95831a1b040aa2f2cb57041856ae8a6651dafbcefce9f912: Status 404 returned error can't find the container with id 2fa0cb2b0b5cbf5d95831a1b040aa2f2cb57041856ae8a6651dafbcefce9f912
Jul 28 17:11:10.972583 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:10.972560    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"]
Jul 28 17:11:10.973193 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:11:10.973174    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0e7ffe4_6286_4ae4_8ad3_81513b136398.slice/crio-4fa003ecba612d37dbf5b47258350798843ced63e5f1cae69ce8a12f4f4dffd2 WatchSource:0}: Error finding container 4fa003ecba612d37dbf5b47258350798843ced63e5f1cae69ce8a12f4f4dffd2: Status 404 returned error can't find the container with id 4fa003ecba612d37dbf5b47258350798843ced63e5f1cae69ce8a12f4f4dffd2
Jul 28 17:11:11.039628 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:11.039572    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4" event={"ID":"ca82ebd1-86d8-450e-bd95-0c50cebff7e0","Type":"ContainerStarted","Data":"2fa0cb2b0b5cbf5d95831a1b040aa2f2cb57041856ae8a6651dafbcefce9f912"}
Jul 28 17:11:11.040643 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:11.040619    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" event={"ID":"f0e7ffe4-6286-4ae4-8ad3-81513b136398","Type":"ContainerStarted","Data":"4fa003ecba612d37dbf5b47258350798843ced63e5f1cae69ce8a12f4f4dffd2"}
Jul 28 17:11:11.041597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:11.041578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb" event={"ID":"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0","Type":"ContainerStarted","Data":"c5cddb72f70143b2b2e65ca1a85af667929f9c415a8b1d56ab4be7c2ac784f6e"}
Jul 28 17:11:16.985068 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:16.985017    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wl2tm"
Jul 28 17:11:17.062120 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.062088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4" event={"ID":"ca82ebd1-86d8-450e-bd95-0c50cebff7e0","Type":"ContainerStarted","Data":"78f3f39ccf345061cf5be9f71dc689f459dab8aabe3debca5ffd370783cd5afe"}
Jul 28 17:11:17.063985 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.063964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" event={"ID":"f0e7ffe4-6286-4ae4-8ad3-81513b136398","Type":"ContainerStarted","Data":"600c20c63a2d1005e106c9ffd5fe959450c6c30b61a879a6d8801eb986cb0180"}
Jul 28 17:11:17.064092 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.064000    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" event={"ID":"f0e7ffe4-6286-4ae4-8ad3-81513b136398","Type":"ContainerStarted","Data":"f40986fab1fe97da972b47f0f084e14e5540d1833ceb37cb27c5caf86f3a43a2"}
Jul 28 17:11:17.064092 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.064016    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" event={"ID":"f0e7ffe4-6286-4ae4-8ad3-81513b136398","Type":"ContainerStarted","Data":"a834ae6fcee2c5f83578365f6c6e27e4df9e72fb850ee7d527ef1faa4b609013"}
Jul 28 17:11:17.065328 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.065303    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb" event={"ID":"c6bbf2f5-cda2-41bb-96e8-aeb3dcf209b0","Type":"ContainerStarted","Data":"06b1a538aa62d76b9e9d9d8d03f52cd3298ee3ed79be00594e7a7f2b79af4f49"}
Jul 28 17:11:17.065504 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.065489    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:17.066954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.066937    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb"
Jul 28 17:11:17.080304 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.080237    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-68c5675bf8-qq5c4" podStartSLOduration=1.31843699 podStartE2EDuration="7.080225636s" totalImagesPullingTime="5.761788646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:17.079545064 +0000 UTC m=+61.839570694" watchObservedRunningTime="2026-07-28 17:11:17.080225636 +0000 UTC m=+61.840251284"
Jul 28 17:11:17.097954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.097919    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c95c6f98b-rwmxb" podStartSLOduration=1.303653879 podStartE2EDuration="7.097908324s" totalImagesPullingTime="5.794254445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:17.09732128 +0000 UTC m=+61.857346908" watchObservedRunningTime="2026-07-28 17:11:17.097908324 +0000 UTC m=+61.857933952"
Jul 28 17:11:17.123923 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:17.123887    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" podStartSLOduration=1.372945592 podStartE2EDuration="7.123877059s" totalImagesPullingTime="5.750931467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:17.122839983 +0000 UTC m=+61.882865626" watchObservedRunningTime="2026-07-28 17:11:17.123877059 +0000 UTC m=+61.883902685"
Jul 28 17:11:21.417817 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.417781    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:11:21.420889 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.420872    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:21.428576 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:21.428558    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:11:21.428644 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:21.428621    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs podName:0b662c0c-d267-40aa-8c94-748caf5b4735 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:25.428608952 +0000 UTC m=+130.188634558 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs") pod "network-metrics-daemon-z4x8n" (UID: "0b662c0c-d267-40aa-8c94-748caf5b4735") : secret "metrics-daemon-secret" not found
Jul 28 17:11:21.518599 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.518571    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:11:21.521443 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.521430    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:21.531280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.531236    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e5b4bddb-be20-4289-bc3b-0b64b94a71b7-original-pull-secret\") pod \"global-pull-secret-syncer-5rqtb\" (UID: \"e5b4bddb-be20-4289-bc3b-0b64b94a71b7\") " pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:11:21.618782 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.618758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:11:21.622090 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.622075    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:21.632574 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.632556    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:21.642357 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.642337    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vk7\" (UniqueName: \"kubernetes.io/projected/41dbd98a-cb15-4862-8ff6-b516258b3076-kube-api-access-x9vk7\") pod \"network-check-target-22bpl\" (UID: \"41dbd98a-cb15-4862-8ff6-b516258b3076\") " pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:11:21.720894 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.720852    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxs5k\""
Jul 28 17:11:21.721913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.721900    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rqtb"
Jul 28 17:11:21.728042 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.728020    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:11:21.859837 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.859810    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rqtb"]
Jul 28 17:11:21.862901 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:11:21.862873    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b4bddb_be20_4289_bc3b_0b64b94a71b7.slice/crio-f6ebcc2495a47e01b10d6ed4c7fb4399eadbd36b28f3cfe4838059ebb32602ae WatchSource:0}: Error finding container f6ebcc2495a47e01b10d6ed4c7fb4399eadbd36b28f3cfe4838059ebb32602ae: Status 404 returned error can't find the container with id f6ebcc2495a47e01b10d6ed4c7fb4399eadbd36b28f3cfe4838059ebb32602ae
Jul 28 17:11:21.878884 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:21.878859    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-22bpl"]
Jul 28 17:11:21.883028 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:11:21.883007    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dbd98a_cb15_4862_8ff6_b516258b3076.slice/crio-8732d335ececd446215ce009e50a9027656335dea3b254bee6d635f71597bdce WatchSource:0}: Error finding container 8732d335ececd446215ce009e50a9027656335dea3b254bee6d635f71597bdce: Status 404 returned error can't find the container with id 8732d335ececd446215ce009e50a9027656335dea3b254bee6d635f71597bdce
Jul 28 17:11:22.020368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:22.020319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:11:22.020368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:22.020363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:11:22.020485 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:22.020470    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:22.020530 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:22.020521    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:54.020508868 +0000 UTC m=+98.780534479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:22.020568 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:22.020473    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:22.020609 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:22.020577    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:11:22.020647 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:22.020633    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:54.020619457 +0000 UTC m=+98.780645063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:11:22.082463 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:22.082441    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-22bpl" event={"ID":"41dbd98a-cb15-4862-8ff6-b516258b3076","Type":"ContainerStarted","Data":"8732d335ececd446215ce009e50a9027656335dea3b254bee6d635f71597bdce"}
Jul 28 17:11:22.083568 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:22.083546    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rqtb" event={"ID":"e5b4bddb-be20-4289-bc3b-0b64b94a71b7","Type":"ContainerStarted","Data":"f6ebcc2495a47e01b10d6ed4c7fb4399eadbd36b28f3cfe4838059ebb32602ae"}
Jul 28 17:11:27.102669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:27.102632    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-22bpl" event={"ID":"41dbd98a-cb15-4862-8ff6-b516258b3076","Type":"ContainerStarted","Data":"d3c899d4f6fb2742e911ca3307d9907ce1857a4b8152f10b0fed2a7a22a4cb30"}
Jul 28 17:11:27.103156 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:27.102709    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:11:27.103983 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:27.103961    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rqtb" event={"ID":"e5b4bddb-be20-4289-bc3b-0b64b94a71b7","Type":"ContainerStarted","Data":"e3b7e48ca42009c26a1e8e491ad1ea76571123ce64798b78f729def81be51cef"}
Jul 28 17:11:27.121187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:27.121152    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-22bpl" podStartSLOduration=66.738029264 podStartE2EDuration="1m11.121141019s" totalImagesPullingTime="4.383111755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:27.120540315 +0000 UTC m=+71.880565943" watchObservedRunningTime="2026-07-28 17:11:27.121141019 +0000 UTC m=+71.881166647"
Jul 28 17:11:27.137961 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:27.137922    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5rqtb" podStartSLOduration=66.730640679 podStartE2EDuration="1m11.137910723s" totalImagesPullingTime="4.407270044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:27.137488993 +0000 UTC m=+71.897514621" watchObservedRunningTime="2026-07-28 17:11:27.137910723 +0000 UTC m=+71.897936351"
Jul 28 17:11:54.038483 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:54.038411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:11:54.038483 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:54.038454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") pod \"image-registry-7458574f8f-q7bxs\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") " pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:11:54.038914 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:54.038547    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:54.038914 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:54.038558    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7458574f8f-q7bxs: secret "image-registry-tls" not found
Jul 28 17:11:54.038914 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:54.038598    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls podName:b5fd7f1f-0140-468c-88e5-c96d27d2a140 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:58.038585373 +0000 UTC m=+162.798610979 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls") pod "image-registry-7458574f8f-q7bxs" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140") : secret "image-registry-tls" not found
Jul 28 17:11:54.038914 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:54.038547    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:54.038914 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:54.038623    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls podName:a5a94646-a349-4508-8f0d-b7511f8c1c44 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:58.038618353 +0000 UTC m=+162.798643959 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls") pod "dns-default-6k8qm" (UID: "a5a94646-a349-4508-8f0d-b7511f8c1c44") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:57.308453 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.308424    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2cwvb"]
Jul 28 17:11:57.351046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.351020    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2cwvb"]
Jul 28 17:11:57.351179 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.351128    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.354503 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.354480    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:11:57.354607 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.354510    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:11:57.354607 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.354510    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:11:57.355774 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.355758    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:11:57.355852 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.355785    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvmsd\""
Jul 28 17:11:57.460549 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.460529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-data-volume\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.460642 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.460568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-crio-socket\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.460642 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.460586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.460642 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.460622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zm8h\" (UniqueName: \"kubernetes.io/projected/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-api-access-9zm8h\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.460736 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.460683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.561933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.561885    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.561933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.561920    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-data-volume\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.562037 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.561948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-crio-socket\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.562037 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.561998    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-crio-socket\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.562037 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:57.562027    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:57.562128 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.562043    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.562128 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:57.562083    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:58.062065829 +0000 UTC m=+102.822091435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:57.562128 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:57.562102    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:57.562128 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.562122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zm8h\" (UniqueName: \"kubernetes.io/projected/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-api-access-9zm8h\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.562303 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:57.562155    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:58.06214167 +0000 UTC m=+102.822167299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:57.562303 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.562239    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-data-volume\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:57.574760 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:57.574733    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zm8h\" (UniqueName: \"kubernetes.io/projected/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-api-access-9zm8h\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:58.065194 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:58.065169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:58.065308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:58.065226    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:58.065360 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:58.065322    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:58.065360 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:58.065336    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:58.065424 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:58.065381    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.065364898 +0000 UTC m=+103.825390504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:58.065424 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:58.065393    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.06538727 +0000 UTC m=+103.825412875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:58.108664 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:58.108641    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-22bpl"
Jul 28 17:11:59.064827 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:59.064804    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hll47_3eb4e5d0-96cd-4008-a7a0-d75e4df4493e/dns-node-resolver/0.log"
Jul 28 17:11:59.073035 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:59.073012    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:59.073126 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:11:59.073060    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:11:59.073126 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:59.073115    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:59.073207 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:59.073149    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:59.073207 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:59.073158    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:12:01.073145638 +0000 UTC m=+105.833171244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:59.073207 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:11:59.073201    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:12:01.073189754 +0000 UTC m=+105.833215361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:00.263778 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:00.263754    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5b8n_6553e69a-e7c5-4d46-b1d1-3ba4a443b68b/node-ca/0.log"
Jul 28 17:12:00.864113 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:00.864084    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jc2sx_b1a87c97-7a97-4e12-8caa-7f2d96d61ffc/serve-healthcheck-canary/0.log"
Jul 28 17:12:01.088700 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:01.088668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:01.088828 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:01.088725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:01.088828 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:01.088801    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:12:01.088898 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:01.088844    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:12:05.088833221 +0000 UTC m=+109.848858829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : configmap "kube-rbac-proxy" not found
Jul 28 17:12:01.088898 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:01.088801    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:01.088967 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:01.088933    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:12:05.088922364 +0000 UTC m=+109.848947971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:05.114502 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:05.114468    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:05.114907 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:05.114524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:05.114907 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:05.114588    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:12:05.114907 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:05.114595    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:05.114907 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:05.114631    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:12:13.114619675 +0000 UTC m=+117.874645282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : configmap "kube-rbac-proxy" not found
Jul 28 17:12:05.114907 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:05.114644    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls podName:43b6fcec-f787-4ab0-aa8e-bb5506a6688b nodeName:}" failed. No retries permitted until 2026-07-28 17:12:13.11463843 +0000 UTC m=+117.874664037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cwvb" (UID: "43b6fcec-f787-4ab0-aa8e-bb5506a6688b") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:13.175518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:13.175487    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:13.175919 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:13.175540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:13.175961 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:13.175948    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:13.177465 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:13.177447    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43b6fcec-f787-4ab0-aa8e-bb5506a6688b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cwvb\" (UID: \"43b6fcec-f787-4ab0-aa8e-bb5506a6688b\") " pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:13.260205 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:13.260179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2cwvb"
Jul 28 17:12:13.378497 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:13.378472    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2cwvb"]
Jul 28 17:12:13.381736 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:12:13.381704    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b6fcec_f787_4ab0_aa8e_bb5506a6688b.slice/crio-5ff04e960e53b17928302b11d16f2253649f7af177da7f051eb4ee2fc1481179 WatchSource:0}: Error finding container 5ff04e960e53b17928302b11d16f2253649f7af177da7f051eb4ee2fc1481179: Status 404 returned error can't find the container with id 5ff04e960e53b17928302b11d16f2253649f7af177da7f051eb4ee2fc1481179
Jul 28 17:12:14.231744 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:14.231706    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cwvb" event={"ID":"43b6fcec-f787-4ab0-aa8e-bb5506a6688b","Type":"ContainerStarted","Data":"5ff04e960e53b17928302b11d16f2253649f7af177da7f051eb4ee2fc1481179"}
Jul 28 17:12:18.245136 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:18.245107    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cwvb" event={"ID":"43b6fcec-f787-4ab0-aa8e-bb5506a6688b","Type":"ContainerStarted","Data":"2346737ce9f2355966637d9142ee9cfaf471f859c14f3b8286ac7d840d32cf1f"}
Jul 28 17:12:22.261180 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:22.261144    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cwvb" event={"ID":"43b6fcec-f787-4ab0-aa8e-bb5506a6688b","Type":"ContainerStarted","Data":"284ff34541c54c8f18d0d2972f4da6491b60e9d2a536a49815c21b37e2cf6830"}
Jul 28 17:12:25.272737 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.272710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cwvb" event={"ID":"43b6fcec-f787-4ab0-aa8e-bb5506a6688b","Type":"ContainerStarted","Data":"a43012b41ca01886c21cca384b89de2cafcd4cad92cedcb1c4aa904b997acc2f"}
Jul 28 17:12:25.294733 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.294693    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2cwvb" podStartSLOduration=17.079877464 podStartE2EDuration="28.294680376s" totalImagesPullingTime="11.214802912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:25.292940436 +0000 UTC m=+130.052966061" watchObservedRunningTime="2026-07-28 17:12:25.294680376 +0000 UTC m=+130.054706003"
Jul 28 17:12:25.472554 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.472529    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:12:25.474516 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.474492    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b662c0c-d267-40aa-8c94-748caf5b4735-metrics-certs\") pod \"network-metrics-daemon-z4x8n\" (UID: \"0b662c0c-d267-40aa-8c94-748caf5b4735\") " pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:12:25.614492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.614421    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sw5vg\""
Jul 28 17:12:25.622734 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.622719    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z4x8n"
Jul 28 17:12:25.744188 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:25.744159    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z4x8n"]
Jul 28 17:12:25.747654 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:12:25.747628    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b662c0c_d267_40aa_8c94_748caf5b4735.slice/crio-7e85c2044b706ddaa908589bee1324f489d7a141a40d95fafc001360a45e46c6 WatchSource:0}: Error finding container 7e85c2044b706ddaa908589bee1324f489d7a141a40d95fafc001360a45e46c6: Status 404 returned error can't find the container with id 7e85c2044b706ddaa908589bee1324f489d7a141a40d95fafc001360a45e46c6
Jul 28 17:12:26.276737 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:26.276699    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z4x8n" event={"ID":"0b662c0c-d267-40aa-8c94-748caf5b4735","Type":"ContainerStarted","Data":"7e85c2044b706ddaa908589bee1324f489d7a141a40d95fafc001360a45e46c6"}
Jul 28 17:12:27.281420 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:27.281389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z4x8n" event={"ID":"0b662c0c-d267-40aa-8c94-748caf5b4735","Type":"ContainerStarted","Data":"d048c1734faf7b4ccc1801967c75104961e04a0983cbc73ee48b8aa1c7d50d27"}
Jul 28 17:12:27.281825 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:27.281436    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z4x8n" event={"ID":"0b662c0c-d267-40aa-8c94-748caf5b4735","Type":"ContainerStarted","Data":"24938230ced8d5a543360ea5279c911a04dca6709eaf01d043370c9ee8f4a34e"}
Jul 28 17:12:27.300855 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:27.300811    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z4x8n" podStartSLOduration=130.420615109 podStartE2EDuration="2m11.300797248s" totalImagesPullingTime="880.182139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:27.300426281 +0000 UTC m=+132.060451908" watchObservedRunningTime="2026-07-28 17:12:27.300797248 +0000 UTC m=+132.060822918"
Jul 28 17:12:36.312774 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.312741    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slc24"]
Jul 28 17:12:36.316003 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.315980    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.320307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.320283    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:12:36.320410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.320389    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44md9\""
Jul 28 17:12:36.320490 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.320407    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:12:36.320490 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.320448    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:12:36.321608 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.321583    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:12:36.321721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.321680    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:12:36.321721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.321696    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:12:36.344370 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344353    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344483 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344376    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2bjz\" (UniqueName: \"kubernetes.io/projected/03f51ace-0d90-439a-b426-9d3e9ce1b000-kube-api-access-w2bjz\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344483 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344394    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-root\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344596 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344492    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-wtmp\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344596 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344543    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-sys\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344596 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344585    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03f51ace-0d90-439a-b426-9d3e9ce1b000-metrics-client-ca\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344705 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-textfile\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344705 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-tls\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.344705 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.344658    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-accelerators-collector-config\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445324 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2bjz\" (UniqueName: \"kubernetes.io/projected/03f51ace-0d90-439a-b426-9d3e9ce1b000-kube-api-access-w2bjz\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445429 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445332    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-root\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445429 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-wtmp\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445429 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445415    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-sys\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03f51ace-0d90-439a-b426-9d3e9ce1b000-metrics-client-ca\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-textfile\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445490    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-root\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-tls\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-accelerators-collector-config\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-sys\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445587 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:36.445587    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:12:36.445910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-wtmp\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.445910 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:36.445648    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-tls podName:03f51ace-0d90-439a-b426-9d3e9ce1b000 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:36.945632256 +0000 UTC m=+141.705657866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-tls") pod "node-exporter-slc24" (UID: "03f51ace-0d90-439a-b426-9d3e9ce1b000") : secret "node-exporter-tls" not found
Jul 28 17:12:36.445910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445810    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-textfile\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.446116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.445961    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03f51ace-0d90-439a-b426-9d3e9ce1b000-metrics-client-ca\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.446116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.446110    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-accelerators-collector-config\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.447772 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.447753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.454145 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.454127    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2bjz\" (UniqueName: \"kubernetes.io/projected/03f51ace-0d90-439a-b426-9d3e9ce1b000-kube-api-access-w2bjz\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.950234 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.950204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-tls\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:36.952366 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:36.952345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03f51ace-0d90-439a-b426-9d3e9ce1b000-node-exporter-tls\") pod \"node-exporter-slc24\" (UID: \"03f51ace-0d90-439a-b426-9d3e9ce1b000\") " pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:37.225068 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:37.225000    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slc24"
Jul 28 17:12:37.234649 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:12:37.234623    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f51ace_0d90_439a_b426_9d3e9ce1b000.slice/crio-ede22b165241d19bfcb3cdca495e6d0fa641a3d4362ccea483b037bc6464d250 WatchSource:0}: Error finding container ede22b165241d19bfcb3cdca495e6d0fa641a3d4362ccea483b037bc6464d250: Status 404 returned error can't find the container with id ede22b165241d19bfcb3cdca495e6d0fa641a3d4362ccea483b037bc6464d250
Jul 28 17:12:37.309693 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:37.309663    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slc24" event={"ID":"03f51ace-0d90-439a-b426-9d3e9ce1b000","Type":"ContainerStarted","Data":"ede22b165241d19bfcb3cdca495e6d0fa641a3d4362ccea483b037bc6464d250"}
Jul 28 17:12:38.314379 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:38.314349    2571 generic.go:356] "Generic (PLEG): container finished" podID="03f51ace-0d90-439a-b426-9d3e9ce1b000" containerID="8d53237df8110427a67c362486ccc11f9893d3d8ddffd16009062f53049535b2" exitCode=0
Jul 28 17:12:38.314678 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:38.314431    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slc24" event={"ID":"03f51ace-0d90-439a-b426-9d3e9ce1b000","Type":"ContainerDied","Data":"8d53237df8110427a67c362486ccc11f9893d3d8ddffd16009062f53049535b2"}
Jul 28 17:12:39.322359 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:39.322324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slc24" event={"ID":"03f51ace-0d90-439a-b426-9d3e9ce1b000","Type":"ContainerStarted","Data":"4882328721f5fd06de6b99bcfd7ea148ae94595ef8e80dd8de3fcc40967fbd28"}
Jul 28 17:12:39.322788 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:39.322416    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slc24" event={"ID":"03f51ace-0d90-439a-b426-9d3e9ce1b000","Type":"ContainerStarted","Data":"91410733a1abba597a679b6382ad401ffd13fff68cd51f287a753981bd476b9d"}
Jul 28 17:12:39.353717 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:39.353672    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slc24" podStartSLOduration=2.452835402 podStartE2EDuration="3.353659181s" totalImagesPullingTime="841.587084ms" totalInitContainerRuntime="59.236695ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:39.351971137 +0000 UTC m=+144.111996763" watchObservedRunningTime="2026-07-28 17:12:39.353659181 +0000 UTC m=+144.113684808"
Jul 28 17:12:40.815049 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:40.815005    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" podUID="f0e7ffe4-6286-4ae4-8ad3-81513b136398" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:12:44.477638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:44.477603    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7458574f8f-q7bxs"]
Jul 28 17:12:44.478071 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:44.477790    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7458574f8f-q7bxs" podUID="b5fd7f1f-0140-468c-88e5-c96d27d2a140"
Jul 28 17:12:45.339186 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.339161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:12:45.347466 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.347447    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:12:45.408378 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408352    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-image-registry-private-configuration\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408508 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408380    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-certificates\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408508 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408404    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-trusted-ca\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408508 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408425    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-installation-pull-secrets\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408508 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408443    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhm22\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-kube-api-access-nhm22\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408508 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408471    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5fd7f1f-0140-468c-88e5-c96d27d2a140-ca-trust-extracted\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408763 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408498    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-bound-sa-token\") pod \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\" (UID: \"b5fd7f1f-0140-468c-88e5-c96d27d2a140\") "
Jul 28 17:12:45.408826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408776    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fd7f1f-0140-468c-88e5-c96d27d2a140-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:12:45.408826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408799    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:45.408919 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.408850    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:45.410688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.410659    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:45.410796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.410688    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:45.410796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.410728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:45.410796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.410775    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-kube-api-access-nhm22" (OuterVolumeSpecName: "kube-api-access-nhm22") pod "b5fd7f1f-0140-468c-88e5-c96d27d2a140" (UID: "b5fd7f1f-0140-468c-88e5-c96d27d2a140"). InnerVolumeSpecName "kube-api-access-nhm22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:45.509057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509028    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-bound-sa-token\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:45.509057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509053    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-image-registry-private-configuration\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:45.509057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509064    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-certificates\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:45.509426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509072    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5fd7f1f-0140-468c-88e5-c96d27d2a140-trusted-ca\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:45.509426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509081    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5fd7f1f-0140-468c-88e5-c96d27d2a140-installation-pull-secrets\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:45.509426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509093    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhm22\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-kube-api-access-nhm22\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:45.509426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:45.509101    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5fd7f1f-0140-468c-88e5-c96d27d2a140-ca-trust-extracted\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:46.342033 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:46.342006    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7458574f8f-q7bxs"
Jul 28 17:12:46.374301 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:46.374256    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7458574f8f-q7bxs"]
Jul 28 17:12:46.378158 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:46.378139    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7458574f8f-q7bxs"]
Jul 28 17:12:46.414737 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:46.414719    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5fd7f1f-0140-468c-88e5-c96d27d2a140-registry-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:12:47.707391 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:47.707361    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5fd7f1f-0140-468c-88e5-c96d27d2a140" path="/var/lib/kubelet/pods/b5fd7f1f-0140-468c-88e5-c96d27d2a140/volumes"
Jul 28 17:12:50.814558 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:50.814520    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" podUID="f0e7ffe4-6286-4ae4-8ad3-81513b136398" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:12:53.177729 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:12:53.177687    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6k8qm" podUID="a5a94646-a349-4508-8f0d-b7511f8c1c44"
Jul 28 17:12:53.361892 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:53.361864    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:12:58.093804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:58.093763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:12:58.095954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:58.095935    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5a94646-a349-4508-8f0d-b7511f8c1c44-metrics-tls\") pod \"dns-default-6k8qm\" (UID: \"a5a94646-a349-4508-8f0d-b7511f8c1c44\") " pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:12:58.165545 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:58.165521    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-947wh\""
Jul 28 17:12:58.173815 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:58.173797    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:12:58.329561 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:58.329511    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6k8qm"]
Jul 28 17:12:58.331875 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:12:58.331849    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a94646_a349_4508_8f0d_b7511f8c1c44.slice/crio-edf15f6f446eb94c43d8a19591cac5b7166ae327ea39e2bf04c6c394ad405034 WatchSource:0}: Error finding container edf15f6f446eb94c43d8a19591cac5b7166ae327ea39e2bf04c6c394ad405034: Status 404 returned error can't find the container with id edf15f6f446eb94c43d8a19591cac5b7166ae327ea39e2bf04c6c394ad405034
Jul 28 17:12:58.376519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:12:58.376465    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6k8qm" event={"ID":"a5a94646-a349-4508-8f0d-b7511f8c1c44","Type":"ContainerStarted","Data":"edf15f6f446eb94c43d8a19591cac5b7166ae327ea39e2bf04c6c394ad405034"}
Jul 28 17:13:00.385790 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.385761    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6k8qm" event={"ID":"a5a94646-a349-4508-8f0d-b7511f8c1c44","Type":"ContainerStarted","Data":"e44aaeb4dad98be07fd6635a354c639a32ca83f87e85bcd24b324fca74f66593"}
Jul 28 17:13:00.386330 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.385815    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6k8qm" event={"ID":"a5a94646-a349-4508-8f0d-b7511f8c1c44","Type":"ContainerStarted","Data":"58021798dbf65c28a744c184082f7971e4ecfbd7f8d448da00e34fd24f588c15"}
Jul 28 17:13:00.386330 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.385903    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:13:00.405061 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.405014    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6k8qm" podStartSLOduration=128.901793848 podStartE2EDuration="2m10.404998754s" totalImagesPullingTime="1.503204906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:00.403389196 +0000 UTC m=+165.163414825" watchObservedRunningTime="2026-07-28 17:13:00.404998754 +0000 UTC m=+165.165024381"
Jul 28 17:13:00.815674 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.815641    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" podUID="f0e7ffe4-6286-4ae4-8ad3-81513b136398" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 28 17:13:00.815805 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.815701    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp"
Jul 28 17:13:00.816118 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.816088    2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"600c20c63a2d1005e106c9ffd5fe959450c6c30b61a879a6d8801eb986cb0180"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 28 17:13:00.816187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:00.816135    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" podUID="f0e7ffe4-6286-4ae4-8ad3-81513b136398" containerName="service-proxy" containerID="cri-o://600c20c63a2d1005e106c9ffd5fe959450c6c30b61a879a6d8801eb986cb0180" gracePeriod=30
Jul 28 17:13:01.391053 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:01.391018    2571 generic.go:356] "Generic (PLEG): container finished" podID="f0e7ffe4-6286-4ae4-8ad3-81513b136398" containerID="600c20c63a2d1005e106c9ffd5fe959450c6c30b61a879a6d8801eb986cb0180" exitCode=2
Jul 28 17:13:01.391474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:01.391093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" event={"ID":"f0e7ffe4-6286-4ae4-8ad3-81513b136398","Type":"ContainerDied","Data":"600c20c63a2d1005e106c9ffd5fe959450c6c30b61a879a6d8801eb986cb0180"}
Jul 28 17:13:01.391474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:01.391136    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f6798d56b-rjfvp" event={"ID":"f0e7ffe4-6286-4ae4-8ad3-81513b136398","Type":"ContainerStarted","Data":"9982bf092f06d63fc3b0aa586fc14b02d93b71ae9a19340642b683a08068da84"}
Jul 28 17:13:10.393412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:13:10.393339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6k8qm"
Jul 28 17:15:15.626841 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:15.626816    2571 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:15:58.803626 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.803600    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-7q84k"]
Jul 28 17:15:58.806073 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.806055    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:58.812552 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.812535    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 17:15:58.812878 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.812864    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:15:58.813546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.813532    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-b55fw\""
Jul 28 17:15:58.813820 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.813806    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:15:58.813902 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.813843    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:15:58.822642 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.822625    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-7q84k"]
Jul 28 17:15:58.904583 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.904562    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzccm\" (UniqueName: \"kubernetes.io/projected/02d0886e-b193-4e15-b2a3-c3afc6807404-kube-api-access-hzccm\") pod \"keda-admission-56f9798bb4-7q84k\" (UID: \"02d0886e-b193-4e15-b2a3-c3afc6807404\") " pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:58.904681 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:58.904596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/02d0886e-b193-4e15-b2a3-c3afc6807404-certificates\") pod \"keda-admission-56f9798bb4-7q84k\" (UID: \"02d0886e-b193-4e15-b2a3-c3afc6807404\") " pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:59.005354 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.005334    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzccm\" (UniqueName: \"kubernetes.io/projected/02d0886e-b193-4e15-b2a3-c3afc6807404-kube-api-access-hzccm\") pod \"keda-admission-56f9798bb4-7q84k\" (UID: \"02d0886e-b193-4e15-b2a3-c3afc6807404\") " pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:59.005468 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.005360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/02d0886e-b193-4e15-b2a3-c3afc6807404-certificates\") pod \"keda-admission-56f9798bb4-7q84k\" (UID: \"02d0886e-b193-4e15-b2a3-c3afc6807404\") " pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:59.007615 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.007589    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/02d0886e-b193-4e15-b2a3-c3afc6807404-certificates\") pod \"keda-admission-56f9798bb4-7q84k\" (UID: \"02d0886e-b193-4e15-b2a3-c3afc6807404\") " pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:59.015422 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.015401    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzccm\" (UniqueName: \"kubernetes.io/projected/02d0886e-b193-4e15-b2a3-c3afc6807404-kube-api-access-hzccm\") pod \"keda-admission-56f9798bb4-7q84k\" (UID: \"02d0886e-b193-4e15-b2a3-c3afc6807404\") " pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:59.115693 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.115644    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:15:59.283815 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.283783    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-7q84k"]
Jul 28 17:15:59.289022 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:15:59.288996    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d0886e_b193_4e15_b2a3_c3afc6807404.slice/crio-f74283263346dddb3469c0af909893856c7a76e74ec9f81812b160e1cfe811b6 WatchSource:0}: Error finding container f74283263346dddb3469c0af909893856c7a76e74ec9f81812b160e1cfe811b6: Status 404 returned error can't find the container with id f74283263346dddb3469c0af909893856c7a76e74ec9f81812b160e1cfe811b6
Jul 28 17:15:59.290313 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.290292    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:15:59.913679 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:15:59.913647    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-7q84k" event={"ID":"02d0886e-b193-4e15-b2a3-c3afc6807404","Type":"ContainerStarted","Data":"f74283263346dddb3469c0af909893856c7a76e74ec9f81812b160e1cfe811b6"}
Jul 28 17:16:02.925794 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:16:02.925763    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-7q84k" event={"ID":"02d0886e-b193-4e15-b2a3-c3afc6807404","Type":"ContainerStarted","Data":"8e09db804b664b5aead67a63d884800de28293fd473f9a1637f780c117c8c59c"}
Jul 28 17:16:02.926173 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:16:02.925891    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:16:02.945130 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:16:02.945088    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-7q84k" podStartSLOduration=2.310087596 podStartE2EDuration="4.945076095s" totalImagesPullingTime="2.634988499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:02.944445301 +0000 UTC m=+347.704470930" watchObservedRunningTime="2026-07-28 17:16:02.945076095 +0000 UTC m=+347.705101722"
Jul 28 17:16:23.931104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:16:23.931035    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-7q84k"
Jul 28 17:17:23.221662 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.221631    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jlqx5"]
Jul 28 17:17:23.224552 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.224538    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.231767 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.231746    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 17:17:23.231884 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.231778    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:17:23.231884 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.231866    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xxbbd\""
Jul 28 17:17:23.231884 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.231871    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:17:23.241133 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.241115    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jlqx5"]
Jul 28 17:17:23.327744 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.327714    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8wp\" (UniqueName: \"kubernetes.io/projected/52c23bf1-afb2-4063-a2e3-a7ea22b12218-kube-api-access-sb8wp\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.327866 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.327760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c23bf1-afb2-4063-a2e3-a7ea22b12218-cert\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.428621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.428595    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8wp\" (UniqueName: \"kubernetes.io/projected/52c23bf1-afb2-4063-a2e3-a7ea22b12218-kube-api-access-sb8wp\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.428735 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.428638    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c23bf1-afb2-4063-a2e3-a7ea22b12218-cert\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.428799 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:17:23.428776    2571 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 28 17:17:23.428854 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:17:23.428840    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52c23bf1-afb2-4063-a2e3-a7ea22b12218-cert podName:52c23bf1-afb2-4063-a2e3-a7ea22b12218 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:23.928819796 +0000 UTC m=+428.688845421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/52c23bf1-afb2-4063-a2e3-a7ea22b12218-cert") pod "llmisvc-controller-manager-654c557856-jlqx5" (UID: "52c23bf1-afb2-4063-a2e3-a7ea22b12218") : secret "llmisvc-webhook-server-cert" not found
Jul 28 17:17:23.439649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.439628    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8wp\" (UniqueName: \"kubernetes.io/projected/52c23bf1-afb2-4063-a2e3-a7ea22b12218-kube-api-access-sb8wp\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.931723 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.931702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c23bf1-afb2-4063-a2e3-a7ea22b12218-cert\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:23.933733 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:23.933714    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52c23bf1-afb2-4063-a2e3-a7ea22b12218-cert\") pod \"llmisvc-controller-manager-654c557856-jlqx5\" (UID: \"52c23bf1-afb2-4063-a2e3-a7ea22b12218\") " pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:24.133840 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:24.133813    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:24.258648 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:24.258616    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-jlqx5"]
Jul 28 17:17:24.262782 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:17:24.262750    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod52c23bf1_afb2_4063_a2e3_a7ea22b12218.slice/crio-abcba9a5a7c55d7af5b65592042413f93555e928f38b64dce3f5f58354c7cb1b WatchSource:0}: Error finding container abcba9a5a7c55d7af5b65592042413f93555e928f38b64dce3f5f58354c7cb1b: Status 404 returned error can't find the container with id abcba9a5a7c55d7af5b65592042413f93555e928f38b64dce3f5f58354c7cb1b
Jul 28 17:17:25.167135 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:25.167111    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5" event={"ID":"52c23bf1-afb2-4063-a2e3-a7ea22b12218","Type":"ContainerStarted","Data":"abcba9a5a7c55d7af5b65592042413f93555e928f38b64dce3f5f58354c7cb1b"}
Jul 28 17:17:28.179168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:28.179136    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5" event={"ID":"52c23bf1-afb2-4063-a2e3-a7ea22b12218","Type":"ContainerStarted","Data":"4e468ee5a7da6524457be420e108b6e861cae5040878db03806ac0366679d160"}
Jul 28 17:17:28.179632 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:28.179267    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:17:28.202519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:28.202477    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5" podStartSLOduration=2.017780073 podStartE2EDuration="5.202464427s" totalImagesPullingTime="3.184684354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:28.200390531 +0000 UTC m=+432.960416158" watchObservedRunningTime="2026-07-28 17:17:28.202464427 +0000 UTC m=+432.962490097"
Jul 28 17:17:59.183787 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:17:59.183760    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-jlqx5"
Jul 28 17:18:42.060806 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.060775    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-rss2s"]
Jul 28 17:18:42.062635 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.062618    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.065603 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.065583    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-7dpkj\""
Jul 28 17:18:42.065710 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.065615    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:18:42.075712 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.075691    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-rss2s"]
Jul 28 17:18:42.123435 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.123411    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd449bba-0f9a-435b-8cf6-db43b7f72141-tls-certs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.123544 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.123448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfszs\" (UniqueName: \"kubernetes.io/projected/fd449bba-0f9a-435b-8cf6-db43b7f72141-kube-api-access-sfszs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.224128 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.224106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfszs\" (UniqueName: \"kubernetes.io/projected/fd449bba-0f9a-435b-8cf6-db43b7f72141-kube-api-access-sfszs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.224217 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.224158    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd449bba-0f9a-435b-8cf6-db43b7f72141-tls-certs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.224282 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:18:42.224268    2571 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 17:18:42.224338 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:18:42.224328    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd449bba-0f9a-435b-8cf6-db43b7f72141-tls-certs podName:fd449bba-0f9a-435b-8cf6-db43b7f72141 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:42.7243105 +0000 UTC m=+507.484336108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fd449bba-0f9a-435b-8cf6-db43b7f72141-tls-certs") pod "model-serving-api-7b954ffc95-rss2s" (UID: "fd449bba-0f9a-435b-8cf6-db43b7f72141") : secret "model-serving-api-tls" not found
Jul 28 17:18:42.233954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.233927    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfszs\" (UniqueName: \"kubernetes.io/projected/fd449bba-0f9a-435b-8cf6-db43b7f72141-kube-api-access-sfszs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.726733 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.726710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd449bba-0f9a-435b-8cf6-db43b7f72141-tls-certs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.729063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.729042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd449bba-0f9a-435b-8cf6-db43b7f72141-tls-certs\") pod \"model-serving-api-7b954ffc95-rss2s\" (UID: \"fd449bba-0f9a-435b-8cf6-db43b7f72141\") " pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:42.972352 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:42.972328    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:43.094725 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:43.094698    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-rss2s"]
Jul 28 17:18:43.098257 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:18:43.098222    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd449bba_0f9a_435b_8cf6_db43b7f72141.slice/crio-5eafb535fbabc60ec67555211e2802bdd5360e4016a19fd1695a87428535306d WatchSource:0}: Error finding container 5eafb535fbabc60ec67555211e2802bdd5360e4016a19fd1695a87428535306d: Status 404 returned error can't find the container with id 5eafb535fbabc60ec67555211e2802bdd5360e4016a19fd1695a87428535306d
Jul 28 17:18:43.409902 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:43.409873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-rss2s" event={"ID":"fd449bba-0f9a-435b-8cf6-db43b7f72141","Type":"ContainerStarted","Data":"5eafb535fbabc60ec67555211e2802bdd5360e4016a19fd1695a87428535306d"}
Jul 28 17:18:45.417696 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:45.417665    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-rss2s" event={"ID":"fd449bba-0f9a-435b-8cf6-db43b7f72141","Type":"ContainerStarted","Data":"0fb6075323733c8e0494ba77c94d0534f4bf14bc04e50b752532fe3fa196353f"}
Jul 28 17:18:45.417964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:45.417772    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:18:45.436411 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:45.436370    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-rss2s" podStartSLOduration=1.337025715 podStartE2EDuration="3.436356208s" totalImagesPullingTime="2.099330493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:45.435361346 +0000 UTC m=+510.195386974" watchObservedRunningTime="2026-07-28 17:18:45.436356208 +0000 UTC m=+510.196381835"
Jul 28 17:18:56.424529 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:18:56.424498    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-rss2s"
Jul 28 17:20:19.003405 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.003374    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"]
Jul 28 17:20:19.006452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.006433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.009110 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.009093    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hjb5m\""
Jul 28 17:20:19.009204 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.009093    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 17:20:19.012881 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.012811    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"]
Jul 28 17:20:19.136505 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.136485    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chxsb\" (UniqueName: \"kubernetes.io/projected/29c40b84-d839-4775-8b38-507399c7ae81-kube-api-access-chxsb\") pod \"seaweedfs-tls-custom-6644bf5d6c-n85pb\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.136595 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.136539    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c40b84-d839-4775-8b38-507399c7ae81-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-n85pb\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.236873 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.236851    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c40b84-d839-4775-8b38-507399c7ae81-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-n85pb\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.236960 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.236880    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chxsb\" (UniqueName: \"kubernetes.io/projected/29c40b84-d839-4775-8b38-507399c7ae81-kube-api-access-chxsb\") pod \"seaweedfs-tls-custom-6644bf5d6c-n85pb\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.237156 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.237139    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c40b84-d839-4775-8b38-507399c7ae81-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-n85pb\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.245932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.245916    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chxsb\" (UniqueName: \"kubernetes.io/projected/29c40b84-d839-4775-8b38-507399c7ae81-kube-api-access-chxsb\") pod \"seaweedfs-tls-custom-6644bf5d6c-n85pb\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.315358 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.315313    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:19.437638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.437611    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"]
Jul 28 17:20:19.440493 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:20:19.440469    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c40b84_d839_4775_8b38_507399c7ae81.slice/crio-f0ce087e58676f3369ae9595f4a94b64e37d6a3a144cfc309defcc2235e8ffce WatchSource:0}: Error finding container f0ce087e58676f3369ae9595f4a94b64e37d6a3a144cfc309defcc2235e8ffce: Status 404 returned error can't find the container with id f0ce087e58676f3369ae9595f4a94b64e37d6a3a144cfc309defcc2235e8ffce
Jul 28 17:20:19.711326 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:19.711296    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb" event={"ID":"29c40b84-d839-4775-8b38-507399c7ae81","Type":"ContainerStarted","Data":"f0ce087e58676f3369ae9595f4a94b64e37d6a3a144cfc309defcc2235e8ffce"}
Jul 28 17:20:22.723863 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:22.723788    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb" event={"ID":"29c40b84-d839-4775-8b38-507399c7ae81","Type":"ContainerStarted","Data":"6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915"}
Jul 28 17:20:22.741621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:22.741578    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb" podStartSLOduration=1.78334613 podStartE2EDuration="4.74156431s" totalImagesPullingTime="2.95821818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:22.740745849 +0000 UTC m=+607.500771476" watchObservedRunningTime="2026-07-28 17:20:22.74156431 +0000 UTC m=+607.501589936"
Jul 28 17:20:24.086571 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:24.086543    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"]
Jul 28 17:20:24.729842 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:24.729794    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb" podUID="29c40b84-d839-4775-8b38-507399c7ae81" containerName="seaweedfs-tls-custom" containerID="cri-o://6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915" gracePeriod=30
Jul 28 17:20:25.956368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:25.956344    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:26.083480 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.083416    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chxsb\" (UniqueName: \"kubernetes.io/projected/29c40b84-d839-4775-8b38-507399c7ae81-kube-api-access-chxsb\") pod \"29c40b84-d839-4775-8b38-507399c7ae81\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") "
Jul 28 17:20:26.083605 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.083487    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c40b84-d839-4775-8b38-507399c7ae81-data\") pod \"29c40b84-d839-4775-8b38-507399c7ae81\" (UID: \"29c40b84-d839-4775-8b38-507399c7ae81\") "
Jul 28 17:20:26.084582 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.084560    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c40b84-d839-4775-8b38-507399c7ae81-data" (OuterVolumeSpecName: "data") pod "29c40b84-d839-4775-8b38-507399c7ae81" (UID: "29c40b84-d839-4775-8b38-507399c7ae81"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:20:26.085494 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.085474    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c40b84-d839-4775-8b38-507399c7ae81-kube-api-access-chxsb" (OuterVolumeSpecName: "kube-api-access-chxsb") pod "29c40b84-d839-4775-8b38-507399c7ae81" (UID: "29c40b84-d839-4775-8b38-507399c7ae81"). InnerVolumeSpecName "kube-api-access-chxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:26.183962 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.183936    2571 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c40b84-d839-4775-8b38-507399c7ae81-data\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:20:26.183962 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.183958    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chxsb\" (UniqueName: \"kubernetes.io/projected/29c40b84-d839-4775-8b38-507399c7ae81-kube-api-access-chxsb\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:20:26.736511 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.736486    2571 generic.go:356] "Generic (PLEG): container finished" podID="29c40b84-d839-4775-8b38-507399c7ae81" containerID="6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915" exitCode=0
Jul 28 17:20:26.736636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.736522    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"
Jul 28 17:20:26.736636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.736539    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb" event={"ID":"29c40b84-d839-4775-8b38-507399c7ae81","Type":"ContainerDied","Data":"6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915"}
Jul 28 17:20:26.736636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.736578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb" event={"ID":"29c40b84-d839-4775-8b38-507399c7ae81","Type":"ContainerDied","Data":"f0ce087e58676f3369ae9595f4a94b64e37d6a3a144cfc309defcc2235e8ffce"}
Jul 28 17:20:26.736636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.736616    2571 scope.go:117] "RemoveContainer" containerID="6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915"
Jul 28 17:20:26.745419 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.745383    2571 scope.go:117] "RemoveContainer" containerID="6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915"
Jul 28 17:20:26.745675 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:20:26.745652    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915\": container with ID starting with 6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915 not found: ID does not exist" containerID="6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915"
Jul 28 17:20:26.745737 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.745687    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915"} err="failed to get container status \"6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915\": rpc error: code = NotFound desc = could not find container \"6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915\": container with ID starting with 6e4c34cdd9335222841031d3e5b679239a52fa8a200b07e1f408194b090ae915 not found: ID does not exist"
Jul 28 17:20:26.759237 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.759216    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"]
Jul 28 17:20:26.763205 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:26.763187    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-n85pb"]
Jul 28 17:20:27.706607 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:27.706575    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c40b84-d839-4775-8b38-507399c7ae81" path="/var/lib/kubelet/pods/29c40b84-d839-4775-8b38-507399c7ae81/volumes"
Jul 28 17:20:44.933712 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.933676    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"]
Jul 28 17:20:44.934617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.934597    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c40b84-d839-4775-8b38-507399c7ae81" containerName="seaweedfs-tls-custom"
Jul 28 17:20:44.934749 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.934622    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c40b84-d839-4775-8b38-507399c7ae81" containerName="seaweedfs-tls-custom"
Jul 28 17:20:44.934868 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.934855    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c40b84-d839-4775-8b38-507399c7ae81" containerName="seaweedfs-tls-custom"
Jul 28 17:20:44.939644 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.939617    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:44.942258 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.942225    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 28 17:20:44.942373 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.942267    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 17:20:44.942373 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.942325    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hjb5m\""
Jul 28 17:20:44.947444 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:44.947415    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"]
Jul 28 17:20:45.105361 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.105338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4ada816-f2f6-4d33-a58f-de33105a831d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.105450 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.105363    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6j5b\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-kube-api-access-d6j5b\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.105450 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.105380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.206081 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.206026    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4ada816-f2f6-4d33-a58f-de33105a831d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.206081 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.206057    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6j5b\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-kube-api-access-d6j5b\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.206081 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.206075    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.206219 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:20:45.206152    2571 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 28 17:20:45.206219 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:20:45.206162    2571 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-x69hq: secret "seaweedfs-tls-serving" not found
Jul 28 17:20:45.206301 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:20:45.206225    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-seaweedfs-tls-serving podName:b4ada816-f2f6-4d33-a58f-de33105a831d nodeName:}" failed. No retries permitted until 2026-07-28 17:20:45.706211095 +0000 UTC m=+630.466236701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-x69hq" (UID: "b4ada816-f2f6-4d33-a58f-de33105a831d") : secret "seaweedfs-tls-serving" not found
Jul 28 17:20:45.206448 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.206431    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b4ada816-f2f6-4d33-a58f-de33105a831d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.216865 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.216838    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6j5b\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-kube-api-access-d6j5b\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.708457 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.708433    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.710637 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.710613    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/b4ada816-f2f6-4d33-a58f-de33105a831d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-x69hq\" (UID: \"b4ada816-f2f6-4d33-a58f-de33105a831d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.848986 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.848966    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"
Jul 28 17:20:45.971174 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:45.971153    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-x69hq"]
Jul 28 17:20:45.973706 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:20:45.973676    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ada816_f2f6_4d33_a58f_de33105a831d.slice/crio-c27cf840c210552071ebea20eb1f361f072e224ca65eeff3d20fc30a92978b06 WatchSource:0}: Error finding container c27cf840c210552071ebea20eb1f361f072e224ca65eeff3d20fc30a92978b06: Status 404 returned error can't find the container with id c27cf840c210552071ebea20eb1f361f072e224ca65eeff3d20fc30a92978b06
Jul 28 17:20:46.797747 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:46.797719    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq" event={"ID":"b4ada816-f2f6-4d33-a58f-de33105a831d","Type":"ContainerStarted","Data":"05aa931bd255cb48dff53b7a3f49e24e16fbb93fcd2f9f8770fea1a1b8d887e8"}
Jul 28 17:20:46.797876 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:46.797761    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq" event={"ID":"b4ada816-f2f6-4d33-a58f-de33105a831d","Type":"ContainerStarted","Data":"c27cf840c210552071ebea20eb1f361f072e224ca65eeff3d20fc30a92978b06"}
Jul 28 17:20:46.817193 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:20:46.817150    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-x69hq" podStartSLOduration=2.544871393 podStartE2EDuration="2.817136569s" totalImagesPullingTime="272.265176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:46.816281859 +0000 UTC m=+631.576307487" watchObservedRunningTime="2026-07-28 17:20:46.817136569 +0000 UTC m=+631.577162196"
Jul 28 17:21:14.639957 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.639921    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"]
Jul 28 17:21:14.645187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.644717    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.647481 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.647458    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:21:14.647596 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.647532    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:21:14.647596 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.647549    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:21:14.647804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.647783    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 17:21:14.647879 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.647812    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:21:14.647971 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.647952    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 17:21:14.648593 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.648571    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:21:14.657803 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.657786    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"]
Jul 28 17:21:14.784027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.783999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.784027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.784028    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e847e0a5-2020-4676-850a-82b2f0daeac9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.784168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.784054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e847e0a5-2020-4676-850a-82b2f0daeac9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.784168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.784074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.784168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.784138    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-api-access-hl6jm\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.884455 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.884425    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e847e0a5-2020-4676-850a-82b2f0daeac9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.884535 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.884459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.884535 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.884514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-api-access-hl6jm\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.884634 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.884552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.884634 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.884575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e847e0a5-2020-4676-850a-82b2f0daeac9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.885020 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.885004    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e847e0a5-2020-4676-850a-82b2f0daeac9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.885284 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.885267    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.885366 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.885300    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.887156 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.887129    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e847e0a5-2020-4676-850a-82b2f0daeac9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.893648 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.893602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-api-access-hl6jm\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-zlc5z\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:14.954474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:14.954455    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:15.079526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:15.079499    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"]
Jul 28 17:21:15.082537 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:21:15.082507    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode847e0a5_2020_4676_850a_82b2f0daeac9.slice/crio-3ad00b750ab740acd0dbe7aad6d80f6d301304ae276063c2ac581d8c8626b1c6 WatchSource:0}: Error finding container 3ad00b750ab740acd0dbe7aad6d80f6d301304ae276063c2ac581d8c8626b1c6: Status 404 returned error can't find the container with id 3ad00b750ab740acd0dbe7aad6d80f6d301304ae276063c2ac581d8c8626b1c6
Jul 28 17:21:15.084537 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:15.084521    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:21:15.892894 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:15.892861    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerStarted","Data":"3ad00b750ab740acd0dbe7aad6d80f6d301304ae276063c2ac581d8c8626b1c6"}
Jul 28 17:21:18.329441 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:18.329421    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:21:18.906314 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:18.906277    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerStarted","Data":"5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2"}
Jul 28 17:21:22.920369 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:22.920338    2571 generic.go:356] "Generic (PLEG): container finished" podID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerID="5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2" exitCode=0
Jul 28 17:21:22.920679 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:22.920381    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerDied","Data":"5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2"}
Jul 28 17:21:35.985485 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:35.985443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerStarted","Data":"a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e"}
Jul 28 17:21:37.994053 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:37.994021    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerStarted","Data":"69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5"}
Jul 28 17:21:41.008887 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:41.008852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerStarted","Data":"b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c"}
Jul 28 17:21:41.009443 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:41.008995    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:41.030879 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:41.030840    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podStartSLOduration=3.069461093 podStartE2EDuration="27.030829319s" totalImagesPullingTime="20.235432696s" totalInitContainerRuntime="3.72593553s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:14 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:41.029457919 +0000 UTC m=+685.789483743" watchObservedRunningTime="2026-07-28 17:21:41.030829319 +0000 UTC m=+685.790854946"
Jul 28 17:21:42.013002 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:42.012973    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:42.013373 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:42.013039    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:42.014392 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:42.014362    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:21:42.015061 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:42.015034    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:42.017890 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:42.017874    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:21:43.016062 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:43.016026    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:21:43.016483 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:43.016346    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:44.019385 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:44.019341    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:21:44.019734 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:44.019695    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:54.019955 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:54.019916    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:21:54.020464 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:21:54.020441    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:04.019576 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:04.019530    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:22:04.019979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:04.019957    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:14.019869 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:14.019771    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:22:14.020295 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:14.020204    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:24.020125 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:24.020084    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:22:24.020540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:24.020495    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:34.019681 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:34.019638    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:22:34.020129 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:34.020105    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:44.020347 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:44.020320    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:22:44.020758 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:44.020547    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:22:49.754173 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.754144    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"]
Jul 28 17:22:49.754639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.754410    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" containerID="cri-o://a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e" gracePeriod=30
Jul 28 17:22:49.754639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.754436    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" containerID="cri-o://b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c" gracePeriod=30
Jul 28 17:22:49.754639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.754484    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" containerID="cri-o://69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5" gracePeriod=30
Jul 28 17:22:49.788519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.788495    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"]
Jul 28 17:22:49.790667 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.790654    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:49.793386 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.793365    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 17:22:49.793483 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.793440    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 17:22:49.806762 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.806742    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"]
Jul 28 17:22:49.924767 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.924740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:49.924856 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.924771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn86r\" (UniqueName: \"kubernetes.io/projected/70f92e18-5ad6-4523-bb13-c910000fd729-kube-api-access-qn86r\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:49.924896 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.924864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70f92e18-5ad6-4523-bb13-c910000fd729-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:49.924933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.924900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70f92e18-5ad6-4523-bb13-c910000fd729-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:49.924933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:49.924922    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025212 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025168    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70f92e18-5ad6-4523-bb13-c910000fd729-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025212 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025195    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70f92e18-5ad6-4523-bb13-c910000fd729-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025212 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025411 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025411 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn86r\" (UniqueName: \"kubernetes.io/projected/70f92e18-5ad6-4523-bb13-c910000fd729-kube-api-access-qn86r\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025619 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025598    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70f92e18-5ad6-4523-bb13-c910000fd729-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025869 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025852    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.025959 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.025942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.027586 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.027567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70f92e18-5ad6-4523-bb13-c910000fd729-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.033824 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.033803    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn86r\" (UniqueName: \"kubernetes.io/projected/70f92e18-5ad6-4523-bb13-c910000fd729-kube-api-access-qn86r\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.099673 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.099653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:50.241363 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.241333    2571 generic.go:356] "Generic (PLEG): container finished" podID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerID="69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5" exitCode=2
Jul 28 17:22:50.241513 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.241408    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerDied","Data":"69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5"}
Jul 28 17:22:50.243658 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:50.243566    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"]
Jul 28 17:22:50.246090 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:22:50.246068    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f92e18_5ad6_4523_bb13_c910000fd729.slice/crio-df023b3bdf29bb89dedd0fb634a1efaf4aef7043439d863c5565acf56f3b7fc0 WatchSource:0}: Error finding container df023b3bdf29bb89dedd0fb634a1efaf4aef7043439d863c5565acf56f3b7fc0: Status 404 returned error can't find the container with id df023b3bdf29bb89dedd0fb634a1efaf4aef7043439d863c5565acf56f3b7fc0
Jul 28 17:22:51.246425 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:51.246392    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerStarted","Data":"87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da"}
Jul 28 17:22:51.246770 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:51.246474    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerStarted","Data":"df023b3bdf29bb89dedd0fb634a1efaf4aef7043439d863c5565acf56f3b7fc0"}
Jul 28 17:22:52.013346 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:52.013311    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Jul 28 17:22:54.020070 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:54.020028    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:22:54.020451 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:54.020391    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:54.257529 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:54.257497    2571 generic.go:356] "Generic (PLEG): container finished" podID="70f92e18-5ad6-4523-bb13-c910000fd729" containerID="87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da" exitCode=0
Jul 28 17:22:54.257657 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:54.257570    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerDied","Data":"87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da"}
Jul 28 17:22:54.260024 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:54.260006    2571 generic.go:356] "Generic (PLEG): container finished" podID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerID="a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e" exitCode=0
Jul 28 17:22:54.260097 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:54.260073    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerDied","Data":"a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e"}
Jul 28 17:22:55.265781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.265750    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerStarted","Data":"000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646"}
Jul 28 17:22:55.266188 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.265813    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerStarted","Data":"f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6"}
Jul 28 17:22:55.266188 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.265836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerStarted","Data":"e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66"}
Jul 28 17:22:55.266188 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.266101    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:55.266335 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.266236    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:55.267300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.267273    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:22:55.290138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:55.290098    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podStartSLOduration=2.733032252 podStartE2EDuration="6.290084851s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.557052599s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:55.288828738 +0000 UTC m=+760.048854366" watchObservedRunningTime="2026-07-28 17:22:55.290084851 +0000 UTC m=+760.050110478"
Jul 28 17:22:56.269526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:56.269484    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:22:56.269526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:56.269519    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:22:56.270513 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:56.270490    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:57.012985 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:57.012951    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Jul 28 17:22:57.272929 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:57.272854    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:22:57.273298 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:22:57.273130    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:02.013809 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:02.013770    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Jul 28 17:23:02.014178 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:02.013884    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:23:02.277149 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:02.277077    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:23:02.277648 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:02.277606    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:23:02.277786 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:02.277766    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:04.019346 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:04.019304    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:23:04.019692 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:04.019644    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:07.013827 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:07.013791    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Jul 28 17:23:12.013394 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:12.013356    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Jul 28 17:23:12.278426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:12.278338    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:23:12.278788 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:12.278764    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:14.019828 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:14.019778    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Jul 28 17:23:14.020210 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:14.019920    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:23:14.020210 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:14.020097    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:14.020210 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:14.020183    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:23:17.013369 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:17.013335    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Jul 28 17:23:19.895878 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.895848    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:23:19.917308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917285    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e847e0a5-2020-4676-850a-82b2f0daeac9-proxy-tls\") pod \"e847e0a5-2020-4676-850a-82b2f0daeac9\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") "
Jul 28 17:23:19.917409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917323    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e847e0a5-2020-4676-850a-82b2f0daeac9-kserve-provision-location\") pod \"e847e0a5-2020-4676-850a-82b2f0daeac9\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") "
Jul 28 17:23:19.917409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917347    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-api-access-hl6jm\") pod \"e847e0a5-2020-4676-850a-82b2f0daeac9\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") "
Jul 28 17:23:19.917409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-cabundle-cert\") pod \"e847e0a5-2020-4676-850a-82b2f0daeac9\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") "
Jul 28 17:23:19.917518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917410    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-rbac-proxy-sar-config\") pod \"e847e0a5-2020-4676-850a-82b2f0daeac9\" (UID: \"e847e0a5-2020-4676-850a-82b2f0daeac9\") "
Jul 28 17:23:19.917637 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917602    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e847e0a5-2020-4676-850a-82b2f0daeac9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e847e0a5-2020-4676-850a-82b2f0daeac9" (UID: "e847e0a5-2020-4676-850a-82b2f0daeac9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:23:19.917853 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917823    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e847e0a5-2020-4676-850a-82b2f0daeac9" (UID: "e847e0a5-2020-4676-850a-82b2f0daeac9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:19.917948 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.917834    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e847e0a5-2020-4676-850a-82b2f0daeac9" (UID: "e847e0a5-2020-4676-850a-82b2f0daeac9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:19.919604 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.919582    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e847e0a5-2020-4676-850a-82b2f0daeac9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e847e0a5-2020-4676-850a-82b2f0daeac9" (UID: "e847e0a5-2020-4676-850a-82b2f0daeac9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:23:19.919721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:19.919686    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-api-access-hl6jm" (OuterVolumeSpecName: "kube-api-access-hl6jm") pod "e847e0a5-2020-4676-850a-82b2f0daeac9" (UID: "e847e0a5-2020-4676-850a-82b2f0daeac9"). InnerVolumeSpecName "kube-api-access-hl6jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:23:20.018512 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.018463    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e847e0a5-2020-4676-850a-82b2f0daeac9-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:23:20.018512 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.018486    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e847e0a5-2020-4676-850a-82b2f0daeac9-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:23:20.018512 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.018496    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl6jm\" (UniqueName: \"kubernetes.io/projected/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-api-access-hl6jm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:23:20.018512 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.018504    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:23:20.018512 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.018513    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e847e0a5-2020-4676-850a-82b2f0daeac9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:23:20.352227 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.352169    2571 generic.go:356] "Generic (PLEG): container finished" podID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerID="b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c" exitCode=0
Jul 28 17:23:20.352227 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.352221    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"
Jul 28 17:23:20.352368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.352268    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerDied","Data":"b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c"}
Jul 28 17:23:20.352368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.352332    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z" event={"ID":"e847e0a5-2020-4676-850a-82b2f0daeac9","Type":"ContainerDied","Data":"3ad00b750ab740acd0dbe7aad6d80f6d301304ae276063c2ac581d8c8626b1c6"}
Jul 28 17:23:20.352368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.352363    2571 scope.go:117] "RemoveContainer" containerID="b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c"
Jul 28 17:23:20.360991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.360972    2571 scope.go:117] "RemoveContainer" containerID="69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5"
Jul 28 17:23:20.367819 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.367802    2571 scope.go:117] "RemoveContainer" containerID="a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e"
Jul 28 17:23:20.374049 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.374034    2571 scope.go:117] "RemoveContainer" containerID="5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2"
Jul 28 17:23:20.375142 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.375124    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"]
Jul 28 17:23:20.378907 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.378889    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-zlc5z"]
Jul 28 17:23:20.380692 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.380680    2571 scope.go:117] "RemoveContainer" containerID="b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c"
Jul 28 17:23:20.380938 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:23:20.380921    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c\": container with ID starting with b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c not found: ID does not exist" containerID="b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c"
Jul 28 17:23:20.380987 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.380953    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c"} err="failed to get container status \"b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c\": rpc error: code = NotFound desc = could not find container \"b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c\": container with ID starting with b6ada92548131fd4b31bd38661ea903f8b512973bb2301d82329b8b71a7e647c not found: ID does not exist"
Jul 28 17:23:20.380987 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.380972    2571 scope.go:117] "RemoveContainer" containerID="69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5"
Jul 28 17:23:20.381209 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:23:20.381192    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5\": container with ID starting with 69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5 not found: ID does not exist" containerID="69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5"
Jul 28 17:23:20.381294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.381217    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5"} err="failed to get container status \"69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5\": rpc error: code = NotFound desc = could not find container \"69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5\": container with ID starting with 69e05a63c0a7b65518da8fc04d0409756836d9d99a91ab06dbf8db7c07e98fe5 not found: ID does not exist"
Jul 28 17:23:20.381294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.381240    2571 scope.go:117] "RemoveContainer" containerID="a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e"
Jul 28 17:23:20.381475 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:23:20.381461    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e\": container with ID starting with a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e not found: ID does not exist" containerID="a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e"
Jul 28 17:23:20.381518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.381478    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e"} err="failed to get container status \"a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e\": rpc error: code = NotFound desc = could not find container \"a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e\": container with ID starting with a45f94f737e4517aaec643da4fde739d1e489ba4bc57d7b6f9a1eb78b345234e not found: ID does not exist"
Jul 28 17:23:20.381518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.381492    2571 scope.go:117] "RemoveContainer" containerID="5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2"
Jul 28 17:23:20.381693 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:23:20.381679    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2\": container with ID starting with 5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2 not found: ID does not exist" containerID="5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2"
Jul 28 17:23:20.381736 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:20.381695    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2"} err="failed to get container status \"5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2\": rpc error: code = NotFound desc = could not find container \"5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2\": container with ID starting with 5581a1fe3b96324f56a0b430094398be80f238b4031bfb568c5aeb251fae24d2 not found: ID does not exist"
Jul 28 17:23:21.707189 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:21.707160    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" path="/var/lib/kubelet/pods/e847e0a5-2020-4676-850a-82b2f0daeac9/volumes"
Jul 28 17:23:22.278443 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:22.278408    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:23:22.278871 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:22.278846    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:32.277904 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:32.277867    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:23:32.278362 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:32.278308    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:42.277525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:42.277483    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:23:42.277913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:42.277789    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:52.278063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:52.278026    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:23:52.278523 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:23:52.278500    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:02.278463 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:02.278431    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:02.278878 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:02.278634    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:14.858394 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:14.858364    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"]
Jul 28 17:24:14.858915 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:14.858664    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" containerID="cri-o://e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66" gracePeriod=30
Jul 28 17:24:14.858915 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:14.858689    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" containerID="cri-o://000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646" gracePeriod=30
Jul 28 17:24:14.858915 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:14.858736    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" containerID="cri-o://f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6" gracePeriod=30
Jul 28 17:24:15.539663 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:15.539633    2571 generic.go:356] "Generic (PLEG): container finished" podID="70f92e18-5ad6-4523-bb13-c910000fd729" containerID="f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6" exitCode=2
Jul 28 17:24:15.539807 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:15.539701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerDied","Data":"f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6"}
Jul 28 17:24:17.272978 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:17.272934    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 17:24:18.551094 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:18.551037    2571 generic.go:356] "Generic (PLEG): container finished" podID="70f92e18-5ad6-4523-bb13-c910000fd729" containerID="e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66" exitCode=0
Jul 28 17:24:18.551394 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:18.551106    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerDied","Data":"e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66"}
Jul 28 17:24:22.273863 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:22.273827    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 17:24:22.278185 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:22.278159    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:24:22.278519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:22.278495    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:27.273200 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:27.273159    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 17:24:27.273591 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:27.273333    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:32.273122 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:32.273084    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 17:24:32.278461 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:32.278428    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:24:32.278779 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:32.278757    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:34.997392 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997363    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"]
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997622    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="storage-initializer"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997632    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="storage-initializer"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997644    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997650    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997662    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997667    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997673    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997678    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997720    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kube-rbac-proxy"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997727    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="agent"
Jul 28 17:24:34.997751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:34.997733    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e847e0a5-2020-4676-850a-82b2f0daeac9" containerName="kserve-container"
Jul 28 17:24:35.000822 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.000807    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.003417 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.003393    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 17:24:35.003510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.003423    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 17:24:35.012273 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.012232    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"]
Jul 28 17:24:35.103862 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.103834    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-cabundle-cert\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.103960 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.103868    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a52edf6-e679-448d-b316-7cec18e6af6f-kserve-provision-location\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.103960 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.103905    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a52edf6-e679-448d-b316-7cec18e6af6f-proxy-tls\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.104029 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.103978    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpmn8\" (UniqueName: \"kubernetes.io/projected/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-api-access-mpmn8\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.104029 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.104009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.204958 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.204933    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpmn8\" (UniqueName: \"kubernetes.io/projected/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-api-access-mpmn8\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205047 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.204964    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205047 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.205036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-cabundle-cert\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205113 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.205072    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a52edf6-e679-448d-b316-7cec18e6af6f-kserve-provision-location\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205165 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.205108    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a52edf6-e679-448d-b316-7cec18e6af6f-proxy-tls\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205506 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.205481    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a52edf6-e679-448d-b316-7cec18e6af6f-kserve-provision-location\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205642 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.205626    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-cabundle-cert\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.205690 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.205646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.207412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.207398    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a52edf6-e679-448d-b316-7cec18e6af6f-proxy-tls\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.214401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.214385    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpmn8\" (UniqueName: \"kubernetes.io/projected/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-api-access-mpmn8\") pod \"isvc-logger-predictor-866fbfd695-glrls\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.309675 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.309626    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:35.435807 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.435782    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"]
Jul 28 17:24:35.437961 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:24:35.437937    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a52edf6_e679_448d_b316_7cec18e6af6f.slice/crio-f27f0ca2947ee5cb1dc05c91a5a86dd45b8376a5d08c79ada208b5936509fd2c WatchSource:0}: Error finding container f27f0ca2947ee5cb1dc05c91a5a86dd45b8376a5d08c79ada208b5936509fd2c: Status 404 returned error can't find the container with id f27f0ca2947ee5cb1dc05c91a5a86dd45b8376a5d08c79ada208b5936509fd2c
Jul 28 17:24:35.607171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.607095    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerStarted","Data":"ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2"}
Jul 28 17:24:35.607171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:35.607150    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerStarted","Data":"f27f0ca2947ee5cb1dc05c91a5a86dd45b8376a5d08c79ada208b5936509fd2c"}
Jul 28 17:24:37.273439 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:37.273404    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 17:24:39.621034 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:39.621004    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerID="ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2" exitCode=0
Jul 28 17:24:39.621395 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:39.621081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerDied","Data":"ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2"}
Jul 28 17:24:40.627918 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.627884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerStarted","Data":"97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76"}
Jul 28 17:24:40.628273 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.627960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerStarted","Data":"e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25"}
Jul 28 17:24:40.628273 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.627986    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerStarted","Data":"c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c"}
Jul 28 17:24:40.628347 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.628279    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:40.628435 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.628420    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:40.629299 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.629277    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:24:40.652015 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:40.651974    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podStartSLOduration=3.209619359 podStartE2EDuration="6.651961034s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.442341675s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:40.649827932 +0000 UTC m=+865.409853559" watchObservedRunningTime="2026-07-28 17:24:40.651961034 +0000 UTC m=+865.411986662"
Jul 28 17:24:41.632309 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:41.632282    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:41.632749 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:41.632444    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:24:41.633305 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:41.633278    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:42.273399 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.273363    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Jul 28 17:24:42.277672 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.277639    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:5000: connect: connection refused"
Jul 28 17:24:42.277802 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.277751    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:42.277939 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.277921    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:42.278007 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.277997    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:42.635912 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.635839    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:24:42.636317 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:42.636192    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:44.998673 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:44.998650    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:45.072835 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.072811    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-kube-rbac-proxy-sar-config\") pod \"70f92e18-5ad6-4523-bb13-c910000fd729\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") "
Jul 28 17:24:45.072946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.072840    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70f92e18-5ad6-4523-bb13-c910000fd729-kserve-provision-location\") pod \"70f92e18-5ad6-4523-bb13-c910000fd729\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") "
Jul 28 17:24:45.072946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.072859    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn86r\" (UniqueName: \"kubernetes.io/projected/70f92e18-5ad6-4523-bb13-c910000fd729-kube-api-access-qn86r\") pod \"70f92e18-5ad6-4523-bb13-c910000fd729\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") "
Jul 28 17:24:45.073278 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.072991    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70f92e18-5ad6-4523-bb13-c910000fd729-proxy-tls\") pod \"70f92e18-5ad6-4523-bb13-c910000fd729\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") "
Jul 28 17:24:45.073278 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.073044    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-cabundle-cert\") pod \"70f92e18-5ad6-4523-bb13-c910000fd729\" (UID: \"70f92e18-5ad6-4523-bb13-c910000fd729\") "
Jul 28 17:24:45.073278 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.073161    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f92e18-5ad6-4523-bb13-c910000fd729-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70f92e18-5ad6-4523-bb13-c910000fd729" (UID: "70f92e18-5ad6-4523-bb13-c910000fd729"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:45.073278 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.073201    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70f92e18-5ad6-4523-bb13-c910000fd729" (UID: "70f92e18-5ad6-4523-bb13-c910000fd729"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:45.073278 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.073266    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70f92e18-5ad6-4523-bb13-c910000fd729-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:24:45.073499 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.073474    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70f92e18-5ad6-4523-bb13-c910000fd729" (UID: "70f92e18-5ad6-4523-bb13-c910000fd729"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:45.074896 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.074876    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f92e18-5ad6-4523-bb13-c910000fd729-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70f92e18-5ad6-4523-bb13-c910000fd729" (UID: "70f92e18-5ad6-4523-bb13-c910000fd729"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:45.074896 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.074887    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f92e18-5ad6-4523-bb13-c910000fd729-kube-api-access-qn86r" (OuterVolumeSpecName: "kube-api-access-qn86r") pod "70f92e18-5ad6-4523-bb13-c910000fd729" (UID: "70f92e18-5ad6-4523-bb13-c910000fd729"). InnerVolumeSpecName "kube-api-access-qn86r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:45.173752 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.173732    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70f92e18-5ad6-4523-bb13-c910000fd729-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:24:45.173752 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.173750    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:24:45.173868 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.173760    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70f92e18-5ad6-4523-bb13-c910000fd729-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:24:45.173868 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.173769    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn86r\" (UniqueName: \"kubernetes.io/projected/70f92e18-5ad6-4523-bb13-c910000fd729-kube-api-access-qn86r\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:24:45.646935 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.646909    2571 generic.go:356] "Generic (PLEG): container finished" podID="70f92e18-5ad6-4523-bb13-c910000fd729" containerID="000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646" exitCode=0
Jul 28 17:24:45.647050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.646966    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"
Jul 28 17:24:45.647050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.646992    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerDied","Data":"000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646"}
Jul 28 17:24:45.647168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.647058    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5" event={"ID":"70f92e18-5ad6-4523-bb13-c910000fd729","Type":"ContainerDied","Data":"df023b3bdf29bb89dedd0fb634a1efaf4aef7043439d863c5565acf56f3b7fc0"}
Jul 28 17:24:45.647168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.647089    2571 scope.go:117] "RemoveContainer" containerID="000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646"
Jul 28 17:24:45.655660 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.655645    2571 scope.go:117] "RemoveContainer" containerID="f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6"
Jul 28 17:24:45.662926 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.662908    2571 scope.go:117] "RemoveContainer" containerID="e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66"
Jul 28 17:24:45.668441 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.668421    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"]
Jul 28 17:24:45.669576 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.669559    2571 scope.go:117] "RemoveContainer" containerID="87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da"
Jul 28 17:24:45.673920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.673901    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-vl6k5"]
Jul 28 17:24:45.676605 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.676589    2571 scope.go:117] "RemoveContainer" containerID="000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646"
Jul 28 17:24:45.676857 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:24:45.676839    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646\": container with ID starting with 000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646 not found: ID does not exist" containerID="000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646"
Jul 28 17:24:45.676933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.676868    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646"} err="failed to get container status \"000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646\": rpc error: code = NotFound desc = could not find container \"000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646\": container with ID starting with 000673e68837d68dda1ce2089d353377f179e14a8c1780019a8658e1c1bdd646 not found: ID does not exist"
Jul 28 17:24:45.676933 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.676891    2571 scope.go:117] "RemoveContainer" containerID="f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6"
Jul 28 17:24:45.677128 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:24:45.677112    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6\": container with ID starting with f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6 not found: ID does not exist" containerID="f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6"
Jul 28 17:24:45.677168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.677134    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6"} err="failed to get container status \"f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6\": rpc error: code = NotFound desc = could not find container \"f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6\": container with ID starting with f4af3e5b6dcdfc46de722c3dd392c715b02d90c4827d030054f2c3f7861209f6 not found: ID does not exist"
Jul 28 17:24:45.677168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.677149    2571 scope.go:117] "RemoveContainer" containerID="e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66"
Jul 28 17:24:45.677414 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:24:45.677400    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66\": container with ID starting with e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66 not found: ID does not exist" containerID="e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66"
Jul 28 17:24:45.677467 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.677420    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66"} err="failed to get container status \"e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66\": rpc error: code = NotFound desc = could not find container \"e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66\": container with ID starting with e74b6cf98198aa5f26cc210cc1b5f67c0cf6d7b8db93a3813f7c6311af753f66 not found: ID does not exist"
Jul 28 17:24:45.677467 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.677438    2571 scope.go:117] "RemoveContainer" containerID="87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da"
Jul 28 17:24:45.677651 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:24:45.677634    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da\": container with ID starting with 87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da not found: ID does not exist" containerID="87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da"
Jul 28 17:24:45.677702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.677659    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da"} err="failed to get container status \"87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da\": rpc error: code = NotFound desc = could not find container \"87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da\": container with ID starting with 87ee9f57f6df4dfcf24369229bfa8f0852647ed272103ecfb9978a3bf1f601da not found: ID does not exist"
Jul 28 17:24:45.707033 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:45.707014    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" path="/var/lib/kubelet/pods/70f92e18-5ad6-4523-bb13-c910000fd729/volumes"
Jul 28 17:24:47.640003 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:47.639972    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:24:47.640602 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:47.640571    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:24:47.640823 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:47.640798    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:57.641435 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:57.641395    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:24:57.641925 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:24:57.641825    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:07.641364 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:07.641331    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:25:07.641842 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:07.641820    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:17.640501 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:17.640461    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:25:17.640894 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:17.640798    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:27.641391 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:27.641346    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:25:27.641828 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:27.641750    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:37.641166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:37.641127    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:25:37.641621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:37.641522    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:47.641353 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:47.641325    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:25:47.641785 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:25:47.641414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:26:00.226672 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.226644    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"]
Jul 28 17:26:00.227029 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.226905    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" containerID="cri-o://c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c" gracePeriod=30
Jul 28 17:26:00.227029 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.226964    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" containerID="cri-o://e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25" gracePeriod=30
Jul 28 17:26:00.227029 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.226906    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" containerID="cri-o://97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76" gracePeriod=30
Jul 28 17:26:00.281568 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281546    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"]
Jul 28 17:26:00.281792 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281781    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="storage-initializer"
Jul 28 17:26:00.281838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281794    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="storage-initializer"
Jul 28 17:26:00.281838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281806    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container"
Jul 28 17:26:00.281838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281811    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container"
Jul 28 17:26:00.281838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281819    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy"
Jul 28 17:26:00.281838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281825    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy"
Jul 28 17:26:00.281838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281838    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent"
Jul 28 17:26:00.282002 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281843    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent"
Jul 28 17:26:00.282002 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281883    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="agent"
Jul 28 17:26:00.282002 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281890    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kube-rbac-proxy"
Jul 28 17:26:00.282002 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.281895    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f92e18-5ad6-4523-bb13-c910000fd729" containerName="kserve-container"
Jul 28 17:26:00.285002 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.284989    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.287545 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.287524    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 17:26:00.288018 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.288004    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 17:26:00.295218 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.295200    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"]
Jul 28 17:26:00.375264 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.375208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.375380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.375292    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjds6\" (UniqueName: \"kubernetes.io/projected/a42ace49-8d7d-4a20-b393-264b520fab05-kube-api-access-gjds6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.375380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.375312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42ace49-8d7d-4a20-b393-264b520fab05-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.375380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.375336    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42ace49-8d7d-4a20-b393-264b520fab05-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.375621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.375392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.476224 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476198    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.476351 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjds6\" (UniqueName: \"kubernetes.io/projected/a42ace49-8d7d-4a20-b393-264b520fab05-kube-api-access-gjds6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.476351 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42ace49-8d7d-4a20-b393-264b520fab05-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.476351 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476326    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42ace49-8d7d-4a20-b393-264b520fab05-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.476500 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476360    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.476752 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476707    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42ace49-8d7d-4a20-b393-264b520fab05-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.477012 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.477052 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.476993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.478952 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.478931    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42ace49-8d7d-4a20-b393-264b520fab05-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.487196 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.487172    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjds6\" (UniqueName: \"kubernetes.io/projected/a42ace49-8d7d-4a20-b393-264b520fab05-kube-api-access-gjds6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-jk6xf\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.594163 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.594126    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:26:00.724208 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.724184    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"]
Jul 28 17:26:00.726584 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:26:00.726554    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42ace49_8d7d_4a20_b393_264b520fab05.slice/crio-b8e07a9434a823d092f1f92d1e0e6f45e5b802ef69fa3b2257c871c38cc9714e WatchSource:0}: Error finding container b8e07a9434a823d092f1f92d1e0e6f45e5b802ef69fa3b2257c871c38cc9714e: Status 404 returned error can't find the container with id b8e07a9434a823d092f1f92d1e0e6f45e5b802ef69fa3b2257c871c38cc9714e
Jul 28 17:26:00.899738 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.899701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerStarted","Data":"60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5"}
Jul 28 17:26:00.899889 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.899799    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerStarted","Data":"b8e07a9434a823d092f1f92d1e0e6f45e5b802ef69fa3b2257c871c38cc9714e"}
Jul 28 17:26:00.902297 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.902275    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerID="e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25" exitCode=2
Jul 28 17:26:00.902388 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:00.902337    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerDied","Data":"e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25"}
Jul 28 17:26:02.636137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:02.636100    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 17:26:04.918001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:04.917973    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerID="c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c" exitCode=0
Jul 28 17:26:04.918340 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:04.918024    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerDied","Data":"c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c"}
Jul 28 17:26:07.636128 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:07.636089    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 17:26:07.640513 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:07.640482    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:26:07.640801 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:07.640773    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:26:12.636443 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:12.636406    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 17:26:12.636952 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:12.636603    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:26:17.636041 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:17.636001    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 17:26:17.641386 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:17.641349    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:26:17.641685 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:17.641660    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:26:22.636140 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:22.636096    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 17:26:27.636540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:27.636504    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 28 17:26:27.640849 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:27.640819    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 17:26:27.641004 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:27.640989    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:26:27.641134 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:27.641113    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:26:27.641236 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:27.641222    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:26:30.397428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.397408    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:26:30.476381 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476357    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a52edf6-e679-448d-b316-7cec18e6af6f-kserve-provision-location\") pod \"2a52edf6-e679-448d-b316-7cec18e6af6f\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") "
Jul 28 17:26:30.476502 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476415    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpmn8\" (UniqueName: \"kubernetes.io/projected/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-api-access-mpmn8\") pod \"2a52edf6-e679-448d-b316-7cec18e6af6f\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") "
Jul 28 17:26:30.476502 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476439    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-rbac-proxy-sar-config\") pod \"2a52edf6-e679-448d-b316-7cec18e6af6f\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") "
Jul 28 17:26:30.476502 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476458    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-cabundle-cert\") pod \"2a52edf6-e679-448d-b316-7cec18e6af6f\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") "
Jul 28 17:26:30.476502 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476485    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a52edf6-e679-448d-b316-7cec18e6af6f-proxy-tls\") pod \"2a52edf6-e679-448d-b316-7cec18e6af6f\" (UID: \"2a52edf6-e679-448d-b316-7cec18e6af6f\") "
Jul 28 17:26:30.476708 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476663    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a52edf6-e679-448d-b316-7cec18e6af6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a52edf6-e679-448d-b316-7cec18e6af6f" (UID: "2a52edf6-e679-448d-b316-7cec18e6af6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:30.476819 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476794    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a52edf6-e679-448d-b316-7cec18e6af6f" (UID: "2a52edf6-e679-448d-b316-7cec18e6af6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:30.476871 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.476798    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a52edf6-e679-448d-b316-7cec18e6af6f" (UID: "2a52edf6-e679-448d-b316-7cec18e6af6f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:30.478360 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.478333    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a52edf6-e679-448d-b316-7cec18e6af6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a52edf6-e679-448d-b316-7cec18e6af6f" (UID: "2a52edf6-e679-448d-b316-7cec18e6af6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:30.478457 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.478438    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-api-access-mpmn8" (OuterVolumeSpecName: "kube-api-access-mpmn8") pod "2a52edf6-e679-448d-b316-7cec18e6af6f" (UID: "2a52edf6-e679-448d-b316-7cec18e6af6f"). InnerVolumeSpecName "kube-api-access-mpmn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:30.577554 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.577533    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a52edf6-e679-448d-b316-7cec18e6af6f-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.577554 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.577554    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a52edf6-e679-448d-b316-7cec18e6af6f-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.577669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.577563    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpmn8\" (UniqueName: \"kubernetes.io/projected/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-api-access-mpmn8\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.577669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.577572    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:26:30.577669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:30.577582    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a52edf6-e679-448d-b316-7cec18e6af6f-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:26:31.000679 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.000655    2571 generic.go:356] "Generic (PLEG): container finished" podID="a42ace49-8d7d-4a20-b393-264b520fab05" containerID="60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5" exitCode=0
Jul 28 17:26:31.000787 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.000732    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerDied","Data":"60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5"}
Jul 28 17:26:31.002050 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.002031    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:26:31.002868 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.002843    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerID="97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76" exitCode=0
Jul 28 17:26:31.002977 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.002900    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerDied","Data":"97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76"}
Jul 28 17:26:31.002977 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.002905    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"
Jul 28 17:26:31.002977 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.002959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls" event={"ID":"2a52edf6-e679-448d-b316-7cec18e6af6f","Type":"ContainerDied","Data":"f27f0ca2947ee5cb1dc05c91a5a86dd45b8376a5d08c79ada208b5936509fd2c"}
Jul 28 17:26:31.003079 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.003004    2571 scope.go:117] "RemoveContainer" containerID="97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76"
Jul 28 17:26:31.011782 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.011768    2571 scope.go:117] "RemoveContainer" containerID="e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25"
Jul 28 17:26:31.018449 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.018430    2571 scope.go:117] "RemoveContainer" containerID="c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c"
Jul 28 17:26:31.026806 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.026788    2571 scope.go:117] "RemoveContainer" containerID="ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2"
Jul 28 17:26:31.034221 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034204    2571 scope.go:117] "RemoveContainer" containerID="97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76"
Jul 28 17:26:31.034480 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:26:31.034460    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76\": container with ID starting with 97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76 not found: ID does not exist" containerID="97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76"
Jul 28 17:26:31.034546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034485    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76"} err="failed to get container status \"97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76\": rpc error: code = NotFound desc = could not find container \"97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76\": container with ID starting with 97bab15ea90bdea22db054ed9366d78fc3c98a11a52b8d48a24e6f460b762e76 not found: ID does not exist"
Jul 28 17:26:31.034546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034502    2571 scope.go:117] "RemoveContainer" containerID="e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25"
Jul 28 17:26:31.034736 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:26:31.034720    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25\": container with ID starting with e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25 not found: ID does not exist" containerID="e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25"
Jul 28 17:26:31.034789 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034740    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25"} err="failed to get container status \"e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25\": rpc error: code = NotFound desc = could not find container \"e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25\": container with ID starting with e4aee64cce257220c4b14eeb7875c976771463e885a3f0bf4ea583b65a377e25 not found: ID does not exist"
Jul 28 17:26:31.034789 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034755    2571 scope.go:117] "RemoveContainer" containerID="c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c"
Jul 28 17:26:31.034970 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:26:31.034953    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c\": container with ID starting with c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c not found: ID does not exist" containerID="c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c"
Jul 28 17:26:31.035027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034974    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c"} err="failed to get container status \"c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c\": rpc error: code = NotFound desc = could not find container \"c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c\": container with ID starting with c19f767d8a465ee48fae7bfd95cb7b2b36253ceda9b648d7c95c96ee5c566c4c not found: ID does not exist"
Jul 28 17:26:31.035027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.034988    2571 scope.go:117] "RemoveContainer" containerID="ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2"
Jul 28 17:26:31.035027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.035006    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"]
Jul 28 17:26:31.035189 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:26:31.035172    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2\": container with ID starting with ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2 not found: ID does not exist" containerID="ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2"
Jul 28 17:26:31.035233 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.035206    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2"} err="failed to get container status \"ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2\": rpc error: code = NotFound desc = could not find container \"ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2\": container with ID starting with ac3f8b499f2feb73417b8d0f6321bd4b2e875226ed664bec4615aef9696f53d2 not found: ID does not exist"
Jul 28 17:26:31.036935 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.036917    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-glrls"]
Jul 28 17:26:31.708524 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:26:31.708492    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" path="/var/lib/kubelet/pods/2a52edf6-e679-448d-b316-7cec18e6af6f/volumes"
Jul 28 17:28:31.463763 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:31.463722    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerStarted","Data":"2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6"}
Jul 28 17:28:31.464188 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:31.463833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerStarted","Data":"9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee"}
Jul 28 17:28:31.464188 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:31.463873    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:28:31.492132 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:31.492083    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" podStartSLOduration=2.023468627 podStartE2EDuration="2m31.492069579s" totalImagesPullingTime="1m59.936307615s" totalInitContainerRuntime="29.532293337s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:31.49035184 +0000 UTC m=+1096.250377473" watchObservedRunningTime="2026-07-28 17:28:31.492069579 +0000 UTC m=+1096.252095207"
Jul 28 17:28:32.468389 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:32.468360    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:28:32.469177 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:32.469150    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 17:28:33.471954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:33.471910    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 17:28:38.477426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:38.477396    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:28:38.477900 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:38.477869    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 28 17:28:48.478833 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:48.478801    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:28:56.225155 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:56.225126    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"]
Jul 28 17:28:56.225576 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:56.225347    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container" containerID="cri-o://9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee" gracePeriod=30
Jul 28 17:28:56.225576 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:56.225385    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kube-rbac-proxy" containerID="cri-o://2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6" gracePeriod=30
Jul 28 17:28:56.557119 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:56.557029    2571 generic.go:356] "Generic (PLEG): container finished" podID="a42ace49-8d7d-4a20-b393-264b520fab05" containerID="2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6" exitCode=2
Jul 28 17:28:56.557119 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:56.557098    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerDied","Data":"2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6"}
Jul 28 17:28:57.505374 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.505320    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:28:57.550735 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.550711    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42ace49-8d7d-4a20-b393-264b520fab05-proxy-tls\") pod \"a42ace49-8d7d-4a20-b393-264b520fab05\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") "
Jul 28 17:28:57.550830 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.550761    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42ace49-8d7d-4a20-b393-264b520fab05-kserve-provision-location\") pod \"a42ace49-8d7d-4a20-b393-264b520fab05\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") "
Jul 28 17:28:57.550830 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.550785    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjds6\" (UniqueName: \"kubernetes.io/projected/a42ace49-8d7d-4a20-b393-264b520fab05-kube-api-access-gjds6\") pod \"a42ace49-8d7d-4a20-b393-264b520fab05\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") "
Jul 28 17:28:57.550830 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.550824    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-cabundle-cert\") pod \"a42ace49-8d7d-4a20-b393-264b520fab05\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") "
Jul 28 17:28:57.550960 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.550846    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-kube-rbac-proxy-sar-config\") pod \"a42ace49-8d7d-4a20-b393-264b520fab05\" (UID: \"a42ace49-8d7d-4a20-b393-264b520fab05\") "
Jul 28 17:28:57.551190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.551168    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a42ace49-8d7d-4a20-b393-264b520fab05" (UID: "a42ace49-8d7d-4a20-b393-264b520fab05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:28:57.551190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.551179    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a42ace49-8d7d-4a20-b393-264b520fab05" (UID: "a42ace49-8d7d-4a20-b393-264b520fab05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:28:57.552722 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.552701    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42ace49-8d7d-4a20-b393-264b520fab05-kube-api-access-gjds6" (OuterVolumeSpecName: "kube-api-access-gjds6") pod "a42ace49-8d7d-4a20-b393-264b520fab05" (UID: "a42ace49-8d7d-4a20-b393-264b520fab05"). InnerVolumeSpecName "kube-api-access-gjds6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:28:57.552836 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.552817    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42ace49-8d7d-4a20-b393-264b520fab05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a42ace49-8d7d-4a20-b393-264b520fab05" (UID: "a42ace49-8d7d-4a20-b393-264b520fab05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:28:57.557423 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.557325    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42ace49-8d7d-4a20-b393-264b520fab05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a42ace49-8d7d-4a20-b393-264b520fab05" (UID: "a42ace49-8d7d-4a20-b393-264b520fab05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:28:57.562674 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.562651    2571 generic.go:356] "Generic (PLEG): container finished" podID="a42ace49-8d7d-4a20-b393-264b520fab05" containerID="9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee" exitCode=0
Jul 28 17:28:57.562755 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.562689    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerDied","Data":"9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee"}
Jul 28 17:28:57.562755 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.562698    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"
Jul 28 17:28:57.562755 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.562747    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf" event={"ID":"a42ace49-8d7d-4a20-b393-264b520fab05","Type":"ContainerDied","Data":"b8e07a9434a823d092f1f92d1e0e6f45e5b802ef69fa3b2257c871c38cc9714e"}
Jul 28 17:28:57.562850 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.562782    2571 scope.go:117] "RemoveContainer" containerID="2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6"
Jul 28 17:28:57.570340 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.570323    2571 scope.go:117] "RemoveContainer" containerID="9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee"
Jul 28 17:28:57.576741 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.576726    2571 scope.go:117] "RemoveContainer" containerID="60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5"
Jul 28 17:28:57.583090 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.583052    2571 scope.go:117] "RemoveContainer" containerID="2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6"
Jul 28 17:28:57.583826 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:28:57.583791    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6\": container with ID starting with 2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6 not found: ID does not exist" containerID="2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6"
Jul 28 17:28:57.583925 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.583833    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6"} err="failed to get container status \"2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6\": rpc error: code = NotFound desc = could not find container \"2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6\": container with ID starting with 2d73b252b8acb42e73459ed363f27a967a1bf4431519fe42dc0953b390fcbcb6 not found: ID does not exist"
Jul 28 17:28:57.583925 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.583857    2571 scope.go:117] "RemoveContainer" containerID="9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee"
Jul 28 17:28:57.584033 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.583957    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"]
Jul 28 17:28:57.584260 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:28:57.584221    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee\": container with ID starting with 9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee not found: ID does not exist" containerID="9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee"
Jul 28 17:28:57.584351 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.584273    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee"} err="failed to get container status \"9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee\": rpc error: code = NotFound desc = could not find container \"9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee\": container with ID starting with 9b4d1600f9911decbfa5668c9fb3b712fcd0f686dac0d5bc22551c2e6db033ee not found: ID does not exist"
Jul 28 17:28:57.584351 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.584307    2571 scope.go:117] "RemoveContainer" containerID="60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5"
Jul 28 17:28:57.584702 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:28:57.584674    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5\": container with ID starting with 60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5 not found: ID does not exist" containerID="60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5"
Jul 28 17:28:57.584789 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.584712    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5"} err="failed to get container status \"60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5\": rpc error: code = NotFound desc = could not find container \"60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5\": container with ID starting with 60574f48bc6f56dd50ae9915634addf4a9d9a779ef60fec6aa33161181113cf5 not found: ID does not exist"
Jul 28 17:28:57.587955 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.587933    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-jk6xf"]
Jul 28 17:28:57.651826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.651804    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42ace49-8d7d-4a20-b393-264b520fab05-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:28:57.651826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.651824    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42ace49-8d7d-4a20-b393-264b520fab05-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:28:57.651932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.651833    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjds6\" (UniqueName: \"kubernetes.io/projected/a42ace49-8d7d-4a20-b393-264b520fab05-kube-api-access-gjds6\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:28:57.651932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.651841    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:28:57.651932 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.651850    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42ace49-8d7d-4a20-b393-264b520fab05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:28:57.706314 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:28:57.706284    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" path="/var/lib/kubelet/pods/a42ace49-8d7d-4a20-b393-264b520fab05/volumes"
Jul 28 17:29:01.410402 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410368    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"]
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410695    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410712    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410729    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="storage-initializer"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410736    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="storage-initializer"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410749    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kube-rbac-proxy"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410756    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kube-rbac-proxy"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410764    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410772    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410790    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410797    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410805    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410813    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410822    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="storage-initializer"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410830    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="storage-initializer"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410893    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kube-rbac-proxy"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410906    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kube-rbac-proxy"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410916    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="agent"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410924    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a52edf6-e679-448d-b316-7cec18e6af6f" containerName="kserve-container"
Jul 28 17:29:01.410946 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.410932    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42ace49-8d7d-4a20-b393-264b520fab05" containerName="kserve-container"
Jul 28 17:29:01.434801 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.434776    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"]
Jul 28 17:29:01.434920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.434889    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.437541 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.437524    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:29:01.437914 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.437898    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:29:01.437982 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.437936    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:29:01.437982 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.437904    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:29:01.437982 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.437937    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:29:01.438875 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.438857    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 17:29:01.438965 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.438891    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 17:29:01.474113 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.474093    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6f9368-054b-4d4e-a456-06a028146951-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.474198 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.474119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.474198 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.474139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcb8\" (UniqueName: \"kubernetes.io/projected/0b6f9368-054b-4d4e-a456-06a028146951-kube-api-access-8gcb8\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.474198 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.474156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6f9368-054b-4d4e-a456-06a028146951-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.474348 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.474236    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.574540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.574518    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6f9368-054b-4d4e-a456-06a028146951-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.574617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.574557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.574617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.574593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcb8\" (UniqueName: \"kubernetes.io/projected/0b6f9368-054b-4d4e-a456-06a028146951-kube-api-access-8gcb8\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.574721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.574619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6f9368-054b-4d4e-a456-06a028146951-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.574721 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.574690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.575046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.575027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6f9368-054b-4d4e-a456-06a028146951-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.575274 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.575240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.575416 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.575395    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.576910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.576892    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6f9368-054b-4d4e-a456-06a028146951-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.583055 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.583035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcb8\" (UniqueName: \"kubernetes.io/projected/0b6f9368-054b-4d4e-a456-06a028146951-kube-api-access-8gcb8\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pr8dd\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.744336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.744285    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:01.890656 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:01.890634    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"]
Jul 28 17:29:01.893150 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:29:01.893124    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6f9368_054b_4d4e_a456_06a028146951.slice/crio-2af1e123c05d5a3b4300deeabe57efc3649164cd046d7f097630af18384b92f9 WatchSource:0}: Error finding container 2af1e123c05d5a3b4300deeabe57efc3649164cd046d7f097630af18384b92f9: Status 404 returned error can't find the container with id 2af1e123c05d5a3b4300deeabe57efc3649164cd046d7f097630af18384b92f9
Jul 28 17:29:02.582034 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:02.581999    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerStarted","Data":"a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31"}
Jul 28 17:29:02.582506 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:02.582073    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerStarted","Data":"2af1e123c05d5a3b4300deeabe57efc3649164cd046d7f097630af18384b92f9"}
Jul 28 17:29:22.649363 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:22.649335    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b6f9368-054b-4d4e-a456-06a028146951" containerID="a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31" exitCode=0
Jul 28 17:29:22.649660 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:22.649372    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerDied","Data":"a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31"}
Jul 28 17:29:23.654702 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:23.654669    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerStarted","Data":"4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141"}
Jul 28 17:29:23.655105 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:23.654734    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerStarted","Data":"2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa"}
Jul 28 17:29:23.655105 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:23.654922    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:23.675502 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:23.675457    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" podStartSLOduration=2.507487042 podStartE2EDuration="22.675442868s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.167955826s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:23.673983246 +0000 UTC m=+1148.434008888" watchObservedRunningTime="2026-07-28 17:29:23.675442868 +0000 UTC m=+1148.435468562"
Jul 28 17:29:24.659594 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:24.659566    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:29:30.668668 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:29:30.668643    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:30:53.706990 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:30:53.706961    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:31:04.744414 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:04.744366    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.22:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:31:07.279732 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:07.279695    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"]
Jul 28 17:31:07.280137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:07.279946    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kserve-container" containerID="cri-o://2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa" gracePeriod=30
Jul 28 17:31:07.280137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:07.279978    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kube-rbac-proxy" containerID="cri-o://4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141" gracePeriod=30
Jul 28 17:31:07.979922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:07.979885    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b6f9368-054b-4d4e-a456-06a028146951" containerID="4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141" exitCode=2
Jul 28 17:31:07.980086 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:07.979962    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerDied","Data":"4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141"}
Jul 28 17:31:08.436950 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.436926    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:31:08.500688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.500669    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6f9368-054b-4d4e-a456-06a028146951-kserve-provision-location\") pod \"0b6f9368-054b-4d4e-a456-06a028146951\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") "
Jul 28 17:31:08.500800 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.500701    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6f9368-054b-4d4e-a456-06a028146951-proxy-tls\") pod \"0b6f9368-054b-4d4e-a456-06a028146951\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") "
Jul 28 17:31:08.500800 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.500731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-cabundle-cert\") pod \"0b6f9368-054b-4d4e-a456-06a028146951\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") "
Jul 28 17:31:08.500800 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.500767    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-kube-rbac-proxy-sar-config\") pod \"0b6f9368-054b-4d4e-a456-06a028146951\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") "
Jul 28 17:31:08.500800 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.500792    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gcb8\" (UniqueName: \"kubernetes.io/projected/0b6f9368-054b-4d4e-a456-06a028146951-kube-api-access-8gcb8\") pod \"0b6f9368-054b-4d4e-a456-06a028146951\" (UID: \"0b6f9368-054b-4d4e-a456-06a028146951\") "
Jul 28 17:31:08.501178 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.501147    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b6f9368-054b-4d4e-a456-06a028146951" (UID: "0b6f9368-054b-4d4e-a456-06a028146951"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:31:08.501178 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.501165    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b6f9368-054b-4d4e-a456-06a028146951" (UID: "0b6f9368-054b-4d4e-a456-06a028146951"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:31:08.502653 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.502629    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6f9368-054b-4d4e-a456-06a028146951-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b6f9368-054b-4d4e-a456-06a028146951" (UID: "0b6f9368-054b-4d4e-a456-06a028146951"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:08.502794 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.502780    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6f9368-054b-4d4e-a456-06a028146951-kube-api-access-8gcb8" (OuterVolumeSpecName: "kube-api-access-8gcb8") pod "0b6f9368-054b-4d4e-a456-06a028146951" (UID: "0b6f9368-054b-4d4e-a456-06a028146951"). InnerVolumeSpecName "kube-api-access-8gcb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:08.504816 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.504794    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f9368-054b-4d4e-a456-06a028146951-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b6f9368-054b-4d4e-a456-06a028146951" (UID: "0b6f9368-054b-4d4e-a456-06a028146951"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:08.602008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.601963    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:31:08.602008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.601981    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b6f9368-054b-4d4e-a456-06a028146951-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:31:08.602008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.601990    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gcb8\" (UniqueName: \"kubernetes.io/projected/0b6f9368-054b-4d4e-a456-06a028146951-kube-api-access-8gcb8\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:31:08.602008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.601999    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b6f9368-054b-4d4e-a456-06a028146951-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:31:08.602008 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.602007    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b6f9368-054b-4d4e-a456-06a028146951-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:31:08.986661 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.986639    2571 generic.go:356] "Generic (PLEG): container finished" podID="0b6f9368-054b-4d4e-a456-06a028146951" containerID="2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa" exitCode=0
Jul 28 17:31:08.986773 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.986690    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"
Jul 28 17:31:08.986773 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.986710    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerDied","Data":"2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa"}
Jul 28 17:31:08.986773 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.986753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd" event={"ID":"0b6f9368-054b-4d4e-a456-06a028146951","Type":"ContainerDied","Data":"2af1e123c05d5a3b4300deeabe57efc3649164cd046d7f097630af18384b92f9"}
Jul 28 17:31:08.986875 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.986779    2571 scope.go:117] "RemoveContainer" containerID="4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141"
Jul 28 17:31:08.994558 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:08.994540    2571 scope.go:117] "RemoveContainer" containerID="2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa"
Jul 28 17:31:09.000978 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.000961    2571 scope.go:117] "RemoveContainer" containerID="a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31"
Jul 28 17:31:09.007336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.007317    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"]
Jul 28 17:31:09.007879 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.007867    2571 scope.go:117] "RemoveContainer" containerID="4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141"
Jul 28 17:31:09.008098 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:31:09.008082    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141\": container with ID starting with 4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141 not found: ID does not exist" containerID="4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141"
Jul 28 17:31:09.008144 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.008105    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141"} err="failed to get container status \"4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141\": rpc error: code = NotFound desc = could not find container \"4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141\": container with ID starting with 4342e7c6eb72e735bf15b2ada7a863e93265cffab781e9f7b647d0f3de403141 not found: ID does not exist"
Jul 28 17:31:09.008144 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.008127    2571 scope.go:117] "RemoveContainer" containerID="2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa"
Jul 28 17:31:09.008339 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:31:09.008324    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa\": container with ID starting with 2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa not found: ID does not exist" containerID="2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa"
Jul 28 17:31:09.008388 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.008343    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa"} err="failed to get container status \"2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa\": rpc error: code = NotFound desc = could not find container \"2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa\": container with ID starting with 2d642b22fbe75a639ce519175396a5492f7e951e77a0fdabc4f4ce2a1317b3fa not found: ID does not exist"
Jul 28 17:31:09.008388 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.008355    2571 scope.go:117] "RemoveContainer" containerID="a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31"
Jul 28 17:31:09.008553 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:31:09.008532    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31\": container with ID starting with a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31 not found: ID does not exist" containerID="a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31"
Jul 28 17:31:09.008605 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.008563    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31"} err="failed to get container status \"a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31\": rpc error: code = NotFound desc = could not find container \"a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31\": container with ID starting with a80b91c3e2d115823ae84904bf71fe146930b935eed4dba21ea7e5d711e67a31 not found: ID does not exist"
Jul 28 17:31:09.011503 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.011484    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pr8dd"]
Jul 28 17:31:09.710632 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:09.710604    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6f9368-054b-4d4e-a456-06a028146951" path="/var/lib/kubelet/pods/0b6f9368-054b-4d4e-a456-06a028146951/volumes"
Jul 28 17:31:12.459287 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459259    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"]
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459508    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kserve-container"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459519    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kserve-container"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459535    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="storage-initializer"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459541    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="storage-initializer"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459547    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kube-rbac-proxy"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459552    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kube-rbac-proxy"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459591    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kube-rbac-proxy"
Jul 28 17:31:12.459638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.459603    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6f9368-054b-4d4e-a456-06a028146951" containerName="kserve-container"
Jul 28 17:31:12.462745 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.462728    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.465669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.465642    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:31:12.465775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.465665    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:31:12.465775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.465686    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 17:31:12.465775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.465642    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:31:12.465775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.465648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:31:12.466653 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.466640    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:31:12.466718 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.466642    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 17:31:12.473818 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.473799    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"]
Jul 28 17:31:12.527692 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.527666    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.527781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.527697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.527781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.527723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.527781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.527768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.527893 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.527787    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z84p\" (UniqueName: \"kubernetes.io/projected/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-api-access-2z84p\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.628715 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.628687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.628796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.628719    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.628796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.628750    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.628796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.628784    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.628919 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.628813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z84p\" (UniqueName: \"kubernetes.io/projected/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-api-access-2z84p\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.629154 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.629135    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.629364 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.629346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.629493 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.629475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.630972 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.630954    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.636991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.636968    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z84p\" (UniqueName: \"kubernetes.io/projected/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-api-access-2z84p\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.772291 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.772233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:12.899577 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:12.899546    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"]
Jul 28 17:31:12.902955 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:31:12.902929    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1388343a_0ab4_4e21_9248_9fd7f4a4d1f2.slice/crio-25b6df5dc4bd7ff80070cfaf2a1809d1c93e8fcdf6815a5d0e1df0927ffb66bb WatchSource:0}: Error finding container 25b6df5dc4bd7ff80070cfaf2a1809d1c93e8fcdf6815a5d0e1df0927ffb66bb: Status 404 returned error can't find the container with id 25b6df5dc4bd7ff80070cfaf2a1809d1c93e8fcdf6815a5d0e1df0927ffb66bb
Jul 28 17:31:13.004057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:13.004023    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerStarted","Data":"f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274"}
Jul 28 17:31:13.004190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:13.004093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerStarted","Data":"25b6df5dc4bd7ff80070cfaf2a1809d1c93e8fcdf6815a5d0e1df0927ffb66bb"}
Jul 28 17:31:42.102665 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:42.102599    2571 generic.go:356] "Generic (PLEG): container finished" podID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerID="f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274" exitCode=0
Jul 28 17:31:42.102665 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:42.102653    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerDied","Data":"f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274"}
Jul 28 17:31:43.108368 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:43.108334    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerStarted","Data":"65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040"}
Jul 28 17:31:43.108775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:43.108394    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerStarted","Data":"b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577"}
Jul 28 17:31:43.108775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:43.108569    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:43.127495 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:43.127452    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" podStartSLOduration=2.380549814 podStartE2EDuration="31.127439222s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.746889408s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:43.126424794 +0000 UTC m=+1287.886450443" watchObservedRunningTime="2026-07-28 17:31:43.127439222 +0000 UTC m=+1287.887464853"
Jul 28 17:31:44.113123 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:44.113094    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:31:50.122782 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:31:50.122754    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:33:21.707695 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:21.707666    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:33:28.681296 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:28.681241    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"]
Jul 28 17:33:28.681676 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:28.681562    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kserve-container" containerID="cri-o://b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577" gracePeriod=30
Jul 28 17:33:28.681676 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:28.681571    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kube-rbac-proxy" containerID="cri-o://65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040" gracePeriod=30
Jul 28 17:33:29.447468 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.447434    2571 generic.go:356] "Generic (PLEG): container finished" podID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerID="65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040" exitCode=2
Jul 28 17:33:29.447638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.447506    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerDied","Data":"65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040"}
Jul 28 17:33:29.916462 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.916440    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:33:29.935174 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.935143    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-proxy-tls\") pod \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") "
Jul 28 17:33:29.935294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.935181    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-cabundle-cert\") pod \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") "
Jul 28 17:33:29.935294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.935222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z84p\" (UniqueName: \"kubernetes.io/projected/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-api-access-2z84p\") pod \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") "
Jul 28 17:33:29.935294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.935272    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kserve-provision-location\") pod \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") "
Jul 28 17:33:29.935806 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.935299    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-rbac-proxy-sar-config\") pod \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\" (UID: \"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2\") "
Jul 28 17:33:29.936595 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.936556    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" (UID: "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:29.940131 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.939830    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-api-access-2z84p" (OuterVolumeSpecName: "kube-api-access-2z84p") pod "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" (UID: "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2"). InnerVolumeSpecName "kube-api-access-2z84p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:29.940131 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.939844    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" (UID: "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:29.943742 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.943639    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" (UID: "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:29.946018 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:29.945993    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" (UID: "1388343a-0ab4-4e21-9248-9fd7f4a4d1f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:30.036781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.036721    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:33:30.036781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.036745    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z84p\" (UniqueName: \"kubernetes.io/projected/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-api-access-2z84p\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:33:30.036781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.036755    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:33:30.036781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.036765    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:33:30.036781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.036774    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:33:30.453264 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.453214    2571 generic.go:356] "Generic (PLEG): container finished" podID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerID="b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577" exitCode=0
Jul 28 17:33:30.453422 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.453286    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerDied","Data":"b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577"}
Jul 28 17:33:30.453422 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.453314    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"
Jul 28 17:33:30.453422 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.453389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9" event={"ID":"1388343a-0ab4-4e21-9248-9fd7f4a4d1f2","Type":"ContainerDied","Data":"25b6df5dc4bd7ff80070cfaf2a1809d1c93e8fcdf6815a5d0e1df0927ffb66bb"}
Jul 28 17:33:30.453588 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.453428    2571 scope.go:117] "RemoveContainer" containerID="65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040"
Jul 28 17:33:30.462230 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.462215    2571 scope.go:117] "RemoveContainer" containerID="b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577"
Jul 28 17:33:30.469122 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.469105    2571 scope.go:117] "RemoveContainer" containerID="f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274"
Jul 28 17:33:30.474750 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.474731    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"]
Jul 28 17:33:30.475760 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.475747    2571 scope.go:117] "RemoveContainer" containerID="65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040"
Jul 28 17:33:30.475993 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:33:30.475974    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040\": container with ID starting with 65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040 not found: ID does not exist" containerID="65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040"
Jul 28 17:33:30.476048 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.475999    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040"} err="failed to get container status \"65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040\": rpc error: code = NotFound desc = could not find container \"65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040\": container with ID starting with 65ab673854ad4d5a1ee6cd5f8e2de39fb2213c5d459f1c2bcffe4c5c0b754040 not found: ID does not exist"
Jul 28 17:33:30.476048 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.476016    2571 scope.go:117] "RemoveContainer" containerID="b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577"
Jul 28 17:33:30.476224 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:33:30.476210    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577\": container with ID starting with b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577 not found: ID does not exist" containerID="b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577"
Jul 28 17:33:30.476274 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.476229    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577"} err="failed to get container status \"b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577\": rpc error: code = NotFound desc = could not find container \"b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577\": container with ID starting with b797aa5fb98dac2bc7aebebf9b59a608634e0fb9bf1f23ef487287cfd7d3c577 not found: ID does not exist"
Jul 28 17:33:30.476274 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.476240    2571 scope.go:117] "RemoveContainer" containerID="f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274"
Jul 28 17:33:30.476503 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:33:30.476488    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274\": container with ID starting with f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274 not found: ID does not exist" containerID="f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274"
Jul 28 17:33:30.476552 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.476509    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274"} err="failed to get container status \"f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274\": rpc error: code = NotFound desc = could not find container \"f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274\": container with ID starting with f650c85112b592bab0cdb276e4156b2031c4d793926770772d7ce1ba54cf5274 not found: ID does not exist"
Jul 28 17:33:30.483423 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:30.483405    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-sjxd9"]
Jul 28 17:33:31.706377 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:31.706352    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" path="/var/lib/kubelet/pods/1388343a-0ab4-4e21-9248-9fd7f4a4d1f2/volumes"
Jul 28 17:33:33.853647 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853618    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"]
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853838    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="storage-initializer"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853848    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="storage-initializer"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853865    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kserve-container"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853870    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kserve-container"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853876    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kube-rbac-proxy"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853881    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kube-rbac-proxy"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853917    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kserve-container"
Jul 28 17:33:33.854001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.853923    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1388343a-0ab4-4e21-9248-9fd7f4a4d1f2" containerName="kube-rbac-proxy"
Jul 28 17:33:33.858638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.858622    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:33.861256 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.861221    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:33:33.861387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.861276    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:33:33.861493 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.861476    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:33:33.861532 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.861509    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:33:33.861597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.861481    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 17:33:33.862298 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.862283    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 17:33:33.862407 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.862286    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:33:33.868288 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.868266    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"]
Jul 28 17:33:33.963747 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.963722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:33.963864 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.963773    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:33.963864 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.963846    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvvh\" (UniqueName: \"kubernetes.io/projected/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-api-access-qdvvh\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:33.963954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.963882    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:33.963954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:33.963901    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.064456 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.064426    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.064573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.064474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.064573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.064510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvvh\" (UniqueName: \"kubernetes.io/projected/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-api-access-qdvvh\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.064573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.064541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.064573 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.064564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.064939 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.064920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.065062 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.065042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.065172 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.065158    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.066681 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.066657    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.073018 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.072999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvvh\" (UniqueName: \"kubernetes.io/projected/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-api-access-qdvvh\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.168229 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.168208    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:34.289895 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.289868    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"]
Jul 28 17:33:34.292450 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:33:34.292422    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856d1763_2fa3_40dc_a47d_b6d5276b9fa3.slice/crio-82f74eefbcd53be01077ce3ce0078c869b6cf1f26672d16a7fc4406a24de2c70 WatchSource:0}: Error finding container 82f74eefbcd53be01077ce3ce0078c869b6cf1f26672d16a7fc4406a24de2c70: Status 404 returned error can't find the container with id 82f74eefbcd53be01077ce3ce0078c869b6cf1f26672d16a7fc4406a24de2c70
Jul 28 17:33:34.468617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.468545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerStarted","Data":"489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964"}
Jul 28 17:33:34.468617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:34.468604    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerStarted","Data":"82f74eefbcd53be01077ce3ce0078c869b6cf1f26672d16a7fc4406a24de2c70"}
Jul 28 17:33:54.534545 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:54.534515    2571 generic.go:356] "Generic (PLEG): container finished" podID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerID="489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964" exitCode=0
Jul 28 17:33:54.534864 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:54.534567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerDied","Data":"489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964"}
Jul 28 17:33:55.540804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:55.540769    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerStarted","Data":"4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb"}
Jul 28 17:33:55.541227 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:55.540827    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerStarted","Data":"0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8"}
Jul 28 17:33:55.541227 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:55.541016    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:33:55.561409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:55.561354    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" podStartSLOduration=2.418704478 podStartE2EDuration="22.561340344s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.142635866s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:55.560692474 +0000 UTC m=+1420.320718103" watchObservedRunningTime="2026-07-28 17:33:55.561340344 +0000 UTC m=+1420.321365971"
Jul 28 17:33:56.548994 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:33:56.548954    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:34:02.558304 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:34:02.558275    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:35:24.706181 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:24.706155    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:35:35.743410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:35.743370    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:35:39.730589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:39.730512    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"]
Jul 28 17:35:39.730996 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:39.730722    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kserve-container" containerID="cri-o://0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8" gracePeriod=30
Jul 28 17:35:39.730996 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:39.730780    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kube-rbac-proxy" containerID="cri-o://4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb" gracePeriod=30
Jul 28 17:35:39.881774 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:39.881747    2571 generic.go:356] "Generic (PLEG): container finished" podID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerID="4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb" exitCode=2
Jul 28 17:35:39.881875 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:39.881825    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerDied","Data":"4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb"}
Jul 28 17:35:40.879135 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.879113    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:35:40.886327 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.886301    2571 generic.go:356] "Generic (PLEG): container finished" podID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerID="0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8" exitCode=0
Jul 28 17:35:40.886428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.886342    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"
Jul 28 17:35:40.886428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.886345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerDied","Data":"0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8"}
Jul 28 17:35:40.886500 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.886461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l" event={"ID":"856d1763-2fa3-40dc-a47d-b6d5276b9fa3","Type":"ContainerDied","Data":"82f74eefbcd53be01077ce3ce0078c869b6cf1f26672d16a7fc4406a24de2c70"}
Jul 28 17:35:40.886500 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.886490    2571 scope.go:117] "RemoveContainer" containerID="4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb"
Jul 28 17:35:40.893359 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.893342    2571 scope.go:117] "RemoveContainer" containerID="0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8"
Jul 28 17:35:40.900468 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.900452    2571 scope.go:117] "RemoveContainer" containerID="489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964"
Jul 28 17:35:40.908027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.908008    2571 scope.go:117] "RemoveContainer" containerID="4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb"
Jul 28 17:35:40.908280 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:35:40.908263    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb\": container with ID starting with 4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb not found: ID does not exist" containerID="4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb"
Jul 28 17:35:40.908335 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.908287    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb"} err="failed to get container status \"4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb\": rpc error: code = NotFound desc = could not find container \"4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb\": container with ID starting with 4723d4d87ff66e22e3c2acb9505d3d4ab6d8efe3286ba8cbbdf1de2214da2ceb not found: ID does not exist"
Jul 28 17:35:40.908335 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.908302    2571 scope.go:117] "RemoveContainer" containerID="0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8"
Jul 28 17:35:40.908559 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:35:40.908536    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8\": container with ID starting with 0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8 not found: ID does not exist" containerID="0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8"
Jul 28 17:35:40.908598 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.908570    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8"} err="failed to get container status \"0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8\": rpc error: code = NotFound desc = could not find container \"0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8\": container with ID starting with 0b4df0e97ae4242a9add1a9cc18b9e8deeaf4aec8bb18715ab2b1ef817aee1f8 not found: ID does not exist"
Jul 28 17:35:40.908632 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.908596    2571 scope.go:117] "RemoveContainer" containerID="489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964"
Jul 28 17:35:40.908879 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:35:40.908864    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964\": container with ID starting with 489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964 not found: ID does not exist" containerID="489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964"
Jul 28 17:35:40.908918 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.908885    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964"} err="failed to get container status \"489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964\": rpc error: code = NotFound desc = could not find container \"489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964\": container with ID starting with 489ba097f3368017af8a1d097969879a1463e2bb12a130e7156212207ba45964 not found: ID does not exist"
Jul 28 17:35:40.980469 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980447    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-proxy-tls\") pod \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") "
Jul 28 17:35:40.980567 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980476    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdvvh\" (UniqueName: \"kubernetes.io/projected/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-api-access-qdvvh\") pod \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") "
Jul 28 17:35:40.980567 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980501    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-cabundle-cert\") pod \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") "
Jul 28 17:35:40.980567 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980546    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-rbac-proxy-sar-config\") pod \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") "
Jul 28 17:35:40.980659 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980574    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kserve-provision-location\") pod \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\" (UID: \"856d1763-2fa3-40dc-a47d-b6d5276b9fa3\") "
Jul 28 17:35:40.980913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980886    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "856d1763-2fa3-40dc-a47d-b6d5276b9fa3" (UID: "856d1763-2fa3-40dc-a47d-b6d5276b9fa3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:40.980913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.980895    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "856d1763-2fa3-40dc-a47d-b6d5276b9fa3" (UID: "856d1763-2fa3-40dc-a47d-b6d5276b9fa3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:40.982440 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.982420    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-api-access-qdvvh" (OuterVolumeSpecName: "kube-api-access-qdvvh") pod "856d1763-2fa3-40dc-a47d-b6d5276b9fa3" (UID: "856d1763-2fa3-40dc-a47d-b6d5276b9fa3"). InnerVolumeSpecName "kube-api-access-qdvvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:40.982519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.982465    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "856d1763-2fa3-40dc-a47d-b6d5276b9fa3" (UID: "856d1763-2fa3-40dc-a47d-b6d5276b9fa3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:40.984715 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:40.984666    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "856d1763-2fa3-40dc-a47d-b6d5276b9fa3" (UID: "856d1763-2fa3-40dc-a47d-b6d5276b9fa3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:35:41.081411 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.081383    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:35:41.081411 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.081408    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvvh\" (UniqueName: \"kubernetes.io/projected/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-api-access-qdvvh\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:35:41.081520 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.081419    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:35:41.081520 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.081427    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:35:41.081520 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.081437    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/856d1763-2fa3-40dc-a47d-b6d5276b9fa3-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:35:41.210005 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.209978    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"]
Jul 28 17:35:41.213433 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.213413    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-65p6l"]
Jul 28 17:35:41.707181 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:41.707157    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" path="/var/lib/kubelet/pods/856d1763-2fa3-40dc-a47d-b6d5276b9fa3/volumes"
Jul 28 17:35:44.947160 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947131    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"]
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947387    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="storage-initializer"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947400    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="storage-initializer"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947417    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kube-rbac-proxy"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947423    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kube-rbac-proxy"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947431    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kserve-container"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947437    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kserve-container"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947484    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kserve-container"
Jul 28 17:35:44.947522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.947495    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="856d1763-2fa3-40dc-a47d-b6d5276b9fa3" containerName="kube-rbac-proxy"
Jul 28 17:35:44.952201 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.952185    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:44.954694 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.954670    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 17:35:44.954810 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.954670    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 17:35:44.954810 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.954671    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:35:44.954810 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.954765    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:35:44.955843 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.955826    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:35:44.955951 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.955845    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:35:44.955951 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.955912    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:35:44.961380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:44.961362    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"]
Jul 28 17:35:45.107306 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.107272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcljs\" (UniqueName: \"kubernetes.io/projected/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-api-access-qcljs\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.107401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.107316    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.107401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.107338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.107401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.107361    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d198db-4fbe-4edd-959c-b22ef63bd831-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.107401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.107390    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d198db-4fbe-4edd-959c-b22ef63bd831-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.208476 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.208430    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcljs\" (UniqueName: \"kubernetes.io/projected/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-api-access-qcljs\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.208476 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.208460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.208582 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.208479    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.208582 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.208497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d198db-4fbe-4edd-959c-b22ef63bd831-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.208729 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.208712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d198db-4fbe-4edd-959c-b22ef63bd831-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.209032 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.209005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d198db-4fbe-4edd-959c-b22ef63bd831-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.209138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.209070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.209138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.209127    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.210806 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.210790    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d198db-4fbe-4edd-959c-b22ef63bd831-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.216801 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.216781    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcljs\" (UniqueName: \"kubernetes.io/projected/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-api-access-qcljs\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.262428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.262404    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:45.386384 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.386294    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"]
Jul 28 17:35:45.388924 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:35:45.388896    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d198db_4fbe_4edd_959c_b22ef63bd831.slice/crio-453eaee5ca0a81c86e6f875f87da1012768ea6a3f1424f167c027f1ad49e6ae2 WatchSource:0}: Error finding container 453eaee5ca0a81c86e6f875f87da1012768ea6a3f1424f167c027f1ad49e6ae2: Status 404 returned error can't find the container with id 453eaee5ca0a81c86e6f875f87da1012768ea6a3f1424f167c027f1ad49e6ae2
Jul 28 17:35:45.909962 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.909928    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerStarted","Data":"5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d"}
Jul 28 17:35:45.910116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:45.910008    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerStarted","Data":"453eaee5ca0a81c86e6f875f87da1012768ea6a3f1424f167c027f1ad49e6ae2"}
Jul 28 17:35:53.937206 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:53.937178    2571 generic.go:356] "Generic (PLEG): container finished" podID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerID="5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d" exitCode=0
Jul 28 17:35:53.937527 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:53.937262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerDied","Data":"5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d"}
Jul 28 17:35:54.943066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:54.943029    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerStarted","Data":"17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d"}
Jul 28 17:35:54.943447 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:54.943090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerStarted","Data":"c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7"}
Jul 28 17:35:54.943447 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:54.943335    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:54.966313 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:54.966274    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podStartSLOduration=2.639533032 podStartE2EDuration="10.966241322s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.32670829s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:54.964739936 +0000 UTC m=+1539.724765564" watchObservedRunningTime="2026-07-28 17:35:54.966241322 +0000 UTC m=+1539.726267024"
Jul 28 17:35:55.946161 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:55.946133    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:35:55.947160 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:55.947134    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:35:56.950382 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:35:56.950344    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:36:01.954936 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:01.954906    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:36:01.955558 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:01.955525    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:36:11.955963 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:11.955936    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:36:26.490456 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:26.490425    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"]
Jul 28 17:36:26.490852 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:26.490687    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container" containerID="cri-o://c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7" gracePeriod=30
Jul 28 17:36:26.490852 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:26.490779    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kube-rbac-proxy" containerID="cri-o://17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d" gracePeriod=30
Jul 28 17:36:26.951425 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:26.951383    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 28 17:36:27.054289 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.054237    2571 generic.go:356] "Generic (PLEG): container finished" podID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerID="17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d" exitCode=2
Jul 28 17:36:27.054451 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.054310    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerDied","Data":"17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d"}
Jul 28 17:36:27.932417 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.932396    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:36:27.991745 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.991724    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d198db-4fbe-4edd-959c-b22ef63bd831-kserve-provision-location\") pod \"47d198db-4fbe-4edd-959c-b22ef63bd831\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") "
Jul 28 17:36:27.991856 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.991761    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-cabundle-cert\") pod \"47d198db-4fbe-4edd-959c-b22ef63bd831\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") "
Jul 28 17:36:27.991856 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.991790    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-rbac-proxy-sar-config\") pod \"47d198db-4fbe-4edd-959c-b22ef63bd831\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") "
Jul 28 17:36:27.991856 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.991825    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d198db-4fbe-4edd-959c-b22ef63bd831-proxy-tls\") pod \"47d198db-4fbe-4edd-959c-b22ef63bd831\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") "
Jul 28 17:36:27.991856 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.991847    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcljs\" (UniqueName: \"kubernetes.io/projected/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-api-access-qcljs\") pod \"47d198db-4fbe-4edd-959c-b22ef63bd831\" (UID: \"47d198db-4fbe-4edd-959c-b22ef63bd831\") "
Jul 28 17:36:27.992127 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.992098    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47d198db-4fbe-4edd-959c-b22ef63bd831" (UID: "47d198db-4fbe-4edd-959c-b22ef63bd831"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:27.992212 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.992148    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47d198db-4fbe-4edd-959c-b22ef63bd831" (UID: "47d198db-4fbe-4edd-959c-b22ef63bd831"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:27.992618 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.992596    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d198db-4fbe-4edd-959c-b22ef63bd831-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47d198db-4fbe-4edd-959c-b22ef63bd831" (UID: "47d198db-4fbe-4edd-959c-b22ef63bd831"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:27.993814 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.993786    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d198db-4fbe-4edd-959c-b22ef63bd831-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47d198db-4fbe-4edd-959c-b22ef63bd831" (UID: "47d198db-4fbe-4edd-959c-b22ef63bd831"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:27.993969 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:27.993952    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-api-access-qcljs" (OuterVolumeSpecName: "kube-api-access-qcljs") pod "47d198db-4fbe-4edd-959c-b22ef63bd831" (UID: "47d198db-4fbe-4edd-959c-b22ef63bd831"). InnerVolumeSpecName "kube-api-access-qcljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:28.059000 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.058945    2571 generic.go:356] "Generic (PLEG): container finished" podID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerID="c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7" exitCode=0
Jul 28 17:36:28.059000 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.058986    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"
Jul 28 17:36:28.059137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.059012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerDied","Data":"c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7"}
Jul 28 17:36:28.059137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.059083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4" event={"ID":"47d198db-4fbe-4edd-959c-b22ef63bd831","Type":"ContainerDied","Data":"453eaee5ca0a81c86e6f875f87da1012768ea6a3f1424f167c027f1ad49e6ae2"}
Jul 28 17:36:28.059137 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.059122    2571 scope.go:117] "RemoveContainer" containerID="17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d"
Jul 28 17:36:28.066515 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.066308    2571 scope.go:117] "RemoveContainer" containerID="c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7"
Jul 28 17:36:28.073410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.073397    2571 scope.go:117] "RemoveContainer" containerID="5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d"
Jul 28 17:36:28.079518 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.079504    2571 scope.go:117] "RemoveContainer" containerID="17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d"
Jul 28 17:36:28.079754 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:36:28.079732    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d\": container with ID starting with 17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d not found: ID does not exist" containerID="17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d"
Jul 28 17:36:28.079849 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.079758    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d"} err="failed to get container status \"17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d\": rpc error: code = NotFound desc = could not find container \"17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d\": container with ID starting with 17017f64d7e8aa7d8d0d3153a4e009081bcdd0e7ccf81632fac2feb62472c59d not found: ID does not exist"
Jul 28 17:36:28.079849 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.079775    2571 scope.go:117] "RemoveContainer" containerID="c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7"
Jul 28 17:36:28.079969 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:36:28.079954    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7\": container with ID starting with c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7 not found: ID does not exist" containerID="c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7"
Jul 28 17:36:28.080010 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.079972    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7"} err="failed to get container status \"c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7\": rpc error: code = NotFound desc = could not find container \"c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7\": container with ID starting with c95d0373d8c02ee8c442c4e1b3b76dddaa7739e52781d5b3636536a004d0c3b7 not found: ID does not exist"
Jul 28 17:36:28.080010 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.079984    2571 scope.go:117] "RemoveContainer" containerID="5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d"
Jul 28 17:36:28.080231 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:36:28.080214    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d\": container with ID starting with 5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d not found: ID does not exist" containerID="5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d"
Jul 28 17:36:28.080310 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.080235    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d"} err="failed to get container status \"5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d\": rpc error: code = NotFound desc = could not find container \"5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d\": container with ID starting with 5ff82cee9efb92137b459e31db4a4b49c76e3450c5377affc552395023c0ab8d not found: ID does not exist"
Jul 28 17:36:28.082380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.082359    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"]
Jul 28 17:36:28.084830 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.084811    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-jhkc4"]
Jul 28 17:36:28.092146 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.092128    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:36:28.092226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.092149    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d198db-4fbe-4edd-959c-b22ef63bd831-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:36:28.092226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.092164    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcljs\" (UniqueName: \"kubernetes.io/projected/47d198db-4fbe-4edd-959c-b22ef63bd831-kube-api-access-qcljs\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:36:28.092226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.092176    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d198db-4fbe-4edd-959c-b22ef63bd831-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:36:28.092226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:28.092189    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d198db-4fbe-4edd-959c-b22ef63bd831-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:36:29.706733 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:29.706705    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" path="/var/lib/kubelet/pods/47d198db-4fbe-4edd-959c-b22ef63bd831/volumes"
Jul 28 17:36:31.665995 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.665940    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"]
Jul 28 17:36:31.666452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666332    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container"
Jul 28 17:36:31.666452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666349    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container"
Jul 28 17:36:31.666452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666371    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kube-rbac-proxy"
Jul 28 17:36:31.666452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666379    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kube-rbac-proxy"
Jul 28 17:36:31.666452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666392    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="storage-initializer"
Jul 28 17:36:31.666452 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666400    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="storage-initializer"
Jul 28 17:36:31.666741 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666468    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kserve-container"
Jul 28 17:36:31.666741 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.666477    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d198db-4fbe-4edd-959c-b22ef63bd831" containerName="kube-rbac-proxy"
Jul 28 17:36:31.670942 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.670922    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.673684 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.673664    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 17:36:31.673940 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.673924    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:36:31.674024 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.673927    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 17:36:31.674087 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.674062    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:36:31.675043 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.675023    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:36:31.675043 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.675045    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:36:31.675184 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.675049    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:36:31.679162 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.679140    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"]
Jul 28 17:36:31.712503 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.712484    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rzn\" (UniqueName: \"kubernetes.io/projected/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-api-access-g2rzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.712611 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.712509    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.712611 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.712540    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.712611 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.712568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.712751 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.712629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.813602 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.813581    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.813687 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.813610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.813687 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.813628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.813793 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:36:31.813718    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 17:36:31.813793 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.813738    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rzn\" (UniqueName: \"kubernetes.io/projected/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-api-access-g2rzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.813793 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:36:31.813770    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls podName:0bb7af6f-6bc7-4cff-9d12-83ce7365c922 nodeName:}" failed. No retries permitted until 2026-07-28 17:36:32.31375186 +0000 UTC m=+1577.073777467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" (UID: "0bb7af6f-6bc7-4cff-9d12-83ce7365c922") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 17:36:31.813793 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.813793    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.814147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.814125    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.814345 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.814329    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.814400 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.814356    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:31.822476 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:31.822458    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rzn\" (UniqueName: \"kubernetes.io/projected/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-api-access-g2rzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:32.316384 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:32.316352    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:32.318513 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:32.318492    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:32.580842 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:32.580789    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:32.706852 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:32.706784    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"]
Jul 28 17:36:32.708980 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:36:32.708949    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb7af6f_6bc7_4cff_9d12_83ce7365c922.slice/crio-f0b0998816440d88263d162045545788d97bf23c66a8aa067ff11fd704bd9258 WatchSource:0}: Error finding container f0b0998816440d88263d162045545788d97bf23c66a8aa067ff11fd704bd9258: Status 404 returned error can't find the container with id f0b0998816440d88263d162045545788d97bf23c66a8aa067ff11fd704bd9258
Jul 28 17:36:33.078491 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:33.078457    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerStarted","Data":"b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa"}
Jul 28 17:36:33.078642 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:33.078509    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerStarted","Data":"f0b0998816440d88263d162045545788d97bf23c66a8aa067ff11fd704bd9258"}
Jul 28 17:36:38.095731 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:38.095700    2571 generic.go:356] "Generic (PLEG): container finished" podID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerID="b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa" exitCode=0
Jul 28 17:36:38.096055 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:38.095736    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerDied","Data":"b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa"}
Jul 28 17:36:39.100963 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:39.100929    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerStarted","Data":"f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96"}
Jul 28 17:36:39.101446 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:39.100997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerStarted","Data":"d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9"}
Jul 28 17:36:39.101446 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:39.101332    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:39.101557 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:39.101464    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:39.102941 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:39.102911    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:36:39.122155 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:39.122116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" podStartSLOduration=3.540724874 podStartE2EDuration="8.122104969s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.581380095s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:39.120163487 +0000 UTC m=+1583.880189115" watchObservedRunningTime="2026-07-28 17:36:39.122104969 +0000 UTC m=+1583.882130595"
Jul 28 17:36:40.104981 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:40.104944    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:36:45.109885 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:45.109854    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:36:45.112117 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:45.110332    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:36:55.111395 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:36:55.111365    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:37:06.552812 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:06.552783    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"]
Jul 28 17:37:06.553310 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:06.553024    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container" containerID="cri-o://d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9" gracePeriod=30
Jul 28 17:37:06.553310 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:06.553042    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kube-rbac-proxy" containerID="cri-o://f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96" gracePeriod=30
Jul 28 17:37:07.201441 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.201404    2571 generic.go:356] "Generic (PLEG): container finished" podID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerID="f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96" exitCode=2
Jul 28 17:37:07.201606 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.201473    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerDied","Data":"f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96"}
Jul 28 17:37:07.788580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.788559    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:37:07.857901 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.857848    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-rbac-proxy-sar-config\") pod \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") "
Jul 28 17:37:07.857901 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.857884    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2rzn\" (UniqueName: \"kubernetes.io/projected/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-api-access-g2rzn\") pod \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") "
Jul 28 17:37:07.858065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.857932    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kserve-provision-location\") pod \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") "
Jul 28 17:37:07.858065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.857966    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-cabundle-cert\") pod \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") "
Jul 28 17:37:07.858065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.858027    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls\") pod \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\" (UID: \"0bb7af6f-6bc7-4cff-9d12-83ce7365c922\") "
Jul 28 17:37:07.858259 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.858219    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0bb7af6f-6bc7-4cff-9d12-83ce7365c922" (UID: "0bb7af6f-6bc7-4cff-9d12-83ce7365c922"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:07.858380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.858354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bb7af6f-6bc7-4cff-9d12-83ce7365c922" (UID: "0bb7af6f-6bc7-4cff-9d12-83ce7365c922"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:07.858782 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.858763    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bb7af6f-6bc7-4cff-9d12-83ce7365c922" (UID: "0bb7af6f-6bc7-4cff-9d12-83ce7365c922"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:37:07.859949 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.859929    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-api-access-g2rzn" (OuterVolumeSpecName: "kube-api-access-g2rzn") pod "0bb7af6f-6bc7-4cff-9d12-83ce7365c922" (UID: "0bb7af6f-6bc7-4cff-9d12-83ce7365c922"). InnerVolumeSpecName "kube-api-access-g2rzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:07.860009 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.859943    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bb7af6f-6bc7-4cff-9d12-83ce7365c922" (UID: "0bb7af6f-6bc7-4cff-9d12-83ce7365c922"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:07.958953 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.958932    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:37:07.958953 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.958951    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:37:07.959063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.958961    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:37:07.959063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.958968    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:37:07.959063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:07.958976    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rzn\" (UniqueName: \"kubernetes.io/projected/0bb7af6f-6bc7-4cff-9d12-83ce7365c922-kube-api-access-g2rzn\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:37:08.206826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.206756    2571 generic.go:356] "Generic (PLEG): container finished" podID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerID="d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9" exitCode=0
Jul 28 17:37:08.206826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.206796    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerDied","Data":"d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9"}
Jul 28 17:37:08.206826 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.206810    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"
Jul 28 17:37:08.207034 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.206857    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs" event={"ID":"0bb7af6f-6bc7-4cff-9d12-83ce7365c922","Type":"ContainerDied","Data":"f0b0998816440d88263d162045545788d97bf23c66a8aa067ff11fd704bd9258"}
Jul 28 17:37:08.207034 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.206897    2571 scope.go:117] "RemoveContainer" containerID="f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96"
Jul 28 17:37:08.216162 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.216118    2571 scope.go:117] "RemoveContainer" containerID="d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9"
Jul 28 17:37:08.223171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.223153    2571 scope.go:117] "RemoveContainer" containerID="b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa"
Jul 28 17:37:08.229641 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.229621    2571 scope.go:117] "RemoveContainer" containerID="f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96"
Jul 28 17:37:08.229924 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:08.229904    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96\": container with ID starting with f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96 not found: ID does not exist" containerID="f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96"
Jul 28 17:37:08.229984 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.229930    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96"} err="failed to get container status \"f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96\": rpc error: code = NotFound desc = could not find container \"f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96\": container with ID starting with f63e37bbf9faf70d0f2ffb09ab2cc719ed55ef17ef026feca67e4333a7c92e96 not found: ID does not exist"
Jul 28 17:37:08.229984 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.229946    2571 scope.go:117] "RemoveContainer" containerID="d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9"
Jul 28 17:37:08.230188 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:08.230173    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9\": container with ID starting with d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9 not found: ID does not exist" containerID="d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9"
Jul 28 17:37:08.230270 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.230191    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9"} err="failed to get container status \"d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9\": rpc error: code = NotFound desc = could not find container \"d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9\": container with ID starting with d2981640c0382f2925eaefd2d6ed76972ce6068a01cca380b56f13ed980a4ec9 not found: ID does not exist"
Jul 28 17:37:08.230270 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.230203    2571 scope.go:117] "RemoveContainer" containerID="b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa"
Jul 28 17:37:08.230446 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:08.230429    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa\": container with ID starting with b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa not found: ID does not exist" containerID="b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa"
Jul 28 17:37:08.230492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.230455    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa"} err="failed to get container status \"b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa\": rpc error: code = NotFound desc = could not find container \"b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa\": container with ID starting with b474e07d3135ca8e3c6d7947e3d55db3315b97a629100c7fdab435cd4212a5aa not found: ID does not exist"
Jul 28 17:37:08.230945 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.230929    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"]
Jul 28 17:37:08.233277 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:08.233260    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-mhtqs"]
Jul 28 17:37:09.706587 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:09.706559    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" path="/var/lib/kubelet/pods/0bb7af6f-6bc7-4cff-9d12-83ce7365c922/volumes"
Jul 28 17:37:11.816654 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816626    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"]
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816844    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="storage-initializer"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816854    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="storage-initializer"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816873    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kube-rbac-proxy"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816879    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kube-rbac-proxy"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816886    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816891    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816929    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kube-rbac-proxy"
Jul 28 17:37:11.816991 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.816936    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb7af6f-6bc7-4cff-9d12-83ce7365c922" containerName="kserve-container"
Jul 28 17:37:11.821409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.821390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.824711 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.824693    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:37:11.824854 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.824719    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:37:11.824854 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.824743    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:37:11.824854 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.824772    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 17:37:11.824854 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.824695    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:37:11.825791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.825775    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:37:11.825791 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.825788    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 17:37:11.832327 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.832307    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"]
Jul 28 17:37:11.835712 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.835693    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"]
Jul 28 17:37:11.835823 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.835810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.838440 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.838426    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 17:37:11.849000 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.848964    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"]
Jul 28 17:37:11.884481 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884453    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.884581 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884500    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89304a83-a7ee-4f86-8737-334ac79fbf66-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.884650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.884708 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46qt\" (UniqueName: \"kubernetes.io/projected/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-api-access-n46qt\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.884773 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.884844 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884829    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.884889 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884862    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.884889 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884885    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z898j\" (UniqueName: \"kubernetes.io/projected/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-api-access-z898j\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.884964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.884964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.884955    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.985876 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.985854    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.985967 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.985884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.985967 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.985902    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.985967 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.985918    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z898j\" (UniqueName: \"kubernetes.io/projected/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-api-access-z898j\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.986174 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986152    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.986234 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986215    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.986320 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.986320 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986302    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.986401 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:11.986305    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:37:11.986401 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:11.986376    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 17:37:11.986477 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986329    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89304a83-a7ee-4f86-8737-334ac79fbf66-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.986477 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:11.986420    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls podName:b1e2f6bf-fb50-4be3-9378-62c1870f4584 nodeName:}" failed. No retries permitted until 2026-07-28 17:37:12.486398674 +0000 UTC m=+1617.246424282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls") pod "isvc-canary-create-v2-predictor-5b976d889b-xszcd" (UID: "b1e2f6bf-fb50-4be3-9378-62c1870f4584") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Jul 28 17:37:11.986477 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:37:11.986450    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls podName:89304a83-a7ee-4f86-8737-334ac79fbf66 nodeName:}" failed. No retries permitted until 2026-07-28 17:37:12.486434377 +0000 UTC m=+1617.246459983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls") pod "isvc-canary-create-predictor-594f5f9bd5-svcq7" (UID: "89304a83-a7ee-4f86-8737-334ac79fbf66") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 17:37:11.986614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.986614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46qt\" (UniqueName: \"kubernetes.io/projected/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-api-access-n46qt\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.986614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.986760 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89304a83-a7ee-4f86-8737-334ac79fbf66-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.986760 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:11.986860 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.986819    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.987085 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.987064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.998678 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.998661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z898j\" (UniqueName: \"kubernetes.io/projected/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-api-access-z898j\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:11.998678 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:11.998676    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46qt\" (UniqueName: \"kubernetes.io/projected/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-api-access-n46qt\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:12.490390 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.490362    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:12.490498 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.490406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:12.492670 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.492643    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-svcq7\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:12.492763 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.492688    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-xszcd\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:12.730838 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.730815    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:12.745831 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.745812    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:12.864601 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.864572    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"]
Jul 28 17:37:12.867869 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:37:12.867845    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89304a83_a7ee_4f86_8737_334ac79fbf66.slice/crio-08a83bac9d1735008fe875a6baa703e68f417f83ce9b22fc9be34a838372aac5 WatchSource:0}: Error finding container 08a83bac9d1735008fe875a6baa703e68f417f83ce9b22fc9be34a838372aac5: Status 404 returned error can't find the container with id 08a83bac9d1735008fe875a6baa703e68f417f83ce9b22fc9be34a838372aac5
Jul 28 17:37:12.881070 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:12.881051    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"]
Jul 28 17:37:12.883219 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:37:12.883200    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e2f6bf_fb50_4be3_9378_62c1870f4584.slice/crio-8bcc5c7f31e0e7adeb505b5cd44846a6572bc695060c7ec78a711858c5a7632c WatchSource:0}: Error finding container 8bcc5c7f31e0e7adeb505b5cd44846a6572bc695060c7ec78a711858c5a7632c: Status 404 returned error can't find the container with id 8bcc5c7f31e0e7adeb505b5cd44846a6572bc695060c7ec78a711858c5a7632c
Jul 28 17:37:13.226941 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:13.226907    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerStarted","Data":"1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043"}
Jul 28 17:37:13.227113 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:13.226975    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerStarted","Data":"8bcc5c7f31e0e7adeb505b5cd44846a6572bc695060c7ec78a711858c5a7632c"}
Jul 28 17:37:13.228686 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:13.228662    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerStarted","Data":"a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e"}
Jul 28 17:37:13.228853 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:13.228717    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerStarted","Data":"08a83bac9d1735008fe875a6baa703e68f417f83ce9b22fc9be34a838372aac5"}
Jul 28 17:37:17.242810 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:17.242778    2571 generic.go:356] "Generic (PLEG): container finished" podID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerID="1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043" exitCode=0
Jul 28 17:37:17.243172 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:17.242851    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerDied","Data":"1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043"}
Jul 28 17:37:17.244458 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:17.244422    2571 generic.go:356] "Generic (PLEG): container finished" podID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerID="a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e" exitCode=0
Jul 28 17:37:17.244458 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:17.244457    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerDied","Data":"a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e"}
Jul 28 17:37:18.250374 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.250338    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerStarted","Data":"fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b"}
Jul 28 17:37:18.250827 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.250413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerStarted","Data":"9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500"}
Jul 28 17:37:18.250827 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.250732    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:18.250939 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.250895    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:18.252235 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.252209    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:18.252661 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.252644    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerStarted","Data":"e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b"}
Jul 28 17:37:18.252718 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.252684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerStarted","Data":"b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139"}
Jul 28 17:37:18.252870 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.252848    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:18.271614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.271555    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podStartSLOduration=3.438135019 podStartE2EDuration="7.271542056s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.833407037s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:18.270147093 +0000 UTC m=+1623.030172721" watchObservedRunningTime="2026-07-28 17:37:18.271542056 +0000 UTC m=+1623.031567740"
Jul 28 17:37:18.292558 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:18.292517    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podStartSLOduration=3.663070793 podStartE2EDuration="7.292507103s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.62943631s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:18.29186296 +0000 UTC m=+1623.051888587" watchObservedRunningTime="2026-07-28 17:37:18.292507103 +0000 UTC m=+1623.052532779"
Jul 28 17:37:19.256619 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:19.256576    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:19.256619 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:19.256608    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:19.258047 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:19.258020    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:37:20.260335 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:20.260294    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:37:24.260698 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:24.260671    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:37:24.261262 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:24.261219    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:25.264265 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:25.264220    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:37:25.264837 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:25.264809    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:37:34.261223 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:34.261185    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:35.265185 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:35.265151    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:37:44.261963 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:44.261920    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:45.265423 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:45.265378    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:37:54.261846 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:54.261806    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:55.265733 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:37:55.265695    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:38:04.261356 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:04.261319    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:38:05.264762 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:05.264728    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:38:14.261463 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:14.261424    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:38:15.265264 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:15.265221    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:38:24.261428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:24.261399    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:38:25.265387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:25.265356    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:38:31.924363 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:31.924333    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"]
Jul 28 17:38:31.924719 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:31.924562    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" containerID="cri-o://b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139" gracePeriod=30
Jul 28 17:38:31.924719 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:31.924584    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kube-rbac-proxy" containerID="cri-o://e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b" gracePeriod=30
Jul 28 17:38:31.928597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:31.928572    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"]
Jul 28 17:38:31.928836 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:31.928796    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" containerID="cri-o://9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500" gracePeriod=30
Jul 28 17:38:31.928961 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:31.928874    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kube-rbac-proxy" containerID="cri-o://fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b" gracePeriod=30
Jul 28 17:38:32.511129 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:32.511101    2571 generic.go:356] "Generic (PLEG): container finished" podID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerID="fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b" exitCode=2
Jul 28 17:38:32.511302 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:32.511163    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerDied","Data":"fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b"}
Jul 28 17:38:32.513066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:32.513047    2571 generic.go:356] "Generic (PLEG): container finished" podID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerID="e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b" exitCode=2
Jul 28 17:38:32.513171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:32.513098    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerDied","Data":"e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b"}
Jul 28 17:38:34.257194 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:34.257150    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 28 17:38:34.261843 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:34.261820    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:38:35.260804 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.260766    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 28 17:38:35.265167 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.265138    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:38:35.761948 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.761924    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:38:35.858976 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.858919    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n46qt\" (UniqueName: \"kubernetes.io/projected/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-api-access-n46qt\") pod \"89304a83-a7ee-4f86-8737-334ac79fbf66\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") "
Jul 28 17:38:35.858976 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.858964    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls\") pod \"89304a83-a7ee-4f86-8737-334ac79fbf66\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") "
Jul 28 17:38:35.859187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.859002    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-rbac-proxy-sar-config\") pod \"89304a83-a7ee-4f86-8737-334ac79fbf66\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") "
Jul 28 17:38:35.859187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.859060    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89304a83-a7ee-4f86-8737-334ac79fbf66-kserve-provision-location\") pod \"89304a83-a7ee-4f86-8737-334ac79fbf66\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") "
Jul 28 17:38:35.859187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.859094    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-cabundle-cert\") pod \"89304a83-a7ee-4f86-8737-334ac79fbf66\" (UID: \"89304a83-a7ee-4f86-8737-334ac79fbf66\") "
Jul 28 17:38:35.859555 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.859437    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89304a83-a7ee-4f86-8737-334ac79fbf66" (UID: "89304a83-a7ee-4f86-8737-334ac79fbf66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:35.859555 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.859442    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89304a83-a7ee-4f86-8737-334ac79fbf66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89304a83-a7ee-4f86-8737-334ac79fbf66" (UID: "89304a83-a7ee-4f86-8737-334ac79fbf66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:35.859725 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.859617    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89304a83-a7ee-4f86-8737-334ac79fbf66" (UID: "89304a83-a7ee-4f86-8737-334ac79fbf66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:35.860951 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.860929    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-api-access-n46qt" (OuterVolumeSpecName: "kube-api-access-n46qt") pod "89304a83-a7ee-4f86-8737-334ac79fbf66" (UID: "89304a83-a7ee-4f86-8737-334ac79fbf66"). InnerVolumeSpecName "kube-api-access-n46qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:35.861167 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.861152    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89304a83-a7ee-4f86-8737-334ac79fbf66" (UID: "89304a83-a7ee-4f86-8737-334ac79fbf66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:35.875577 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.875558    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:38:35.959954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.959935    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-cabundle-cert\") pod \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") "
Jul 28 17:38:35.960040 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.959982    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls\") pod \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") "
Jul 28 17:38:35.960040 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960014    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-rbac-proxy-sar-config\") pod \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") "
Jul 28 17:38:35.960116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960052    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kserve-provision-location\") pod \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") "
Jul 28 17:38:35.960158 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960112    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z898j\" (UniqueName: \"kubernetes.io/projected/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-api-access-z898j\") pod \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\" (UID: \"b1e2f6bf-fb50-4be3-9378-62c1870f4584\") "
Jul 28 17:38:35.960327 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960308    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89304a83-a7ee-4f86-8737-334ac79fbf66-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960332    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960343    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89304a83-a7ee-4f86-8737-334ac79fbf66-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960351    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89304a83-a7ee-4f86-8737-334ac79fbf66-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960360    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n46qt\" (UniqueName: \"kubernetes.io/projected/89304a83-a7ee-4f86-8737-334ac79fbf66-kube-api-access-n46qt\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960354    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1e2f6bf-fb50-4be3-9378-62c1870f4584" (UID: "b1e2f6bf-fb50-4be3-9378-62c1870f4584"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960367    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1e2f6bf-fb50-4be3-9378-62c1870f4584" (UID: "b1e2f6bf-fb50-4be3-9378-62c1870f4584"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:35.960410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.960378    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1e2f6bf-fb50-4be3-9378-62c1870f4584" (UID: "b1e2f6bf-fb50-4be3-9378-62c1870f4584"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:35.961811 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.961796    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1e2f6bf-fb50-4be3-9378-62c1870f4584" (UID: "b1e2f6bf-fb50-4be3-9378-62c1870f4584"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:35.961968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:35.961946    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-api-access-z898j" (OuterVolumeSpecName: "kube-api-access-z898j") pod "b1e2f6bf-fb50-4be3-9378-62c1870f4584" (UID: "b1e2f6bf-fb50-4be3-9378-62c1870f4584"). InnerVolumeSpecName "kube-api-access-z898j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:36.061532 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.061512    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1e2f6bf-fb50-4be3-9378-62c1870f4584-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:36.061532 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.061532    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:36.061649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.061541    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:36.061649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.061550    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z898j\" (UniqueName: \"kubernetes.io/projected/b1e2f6bf-fb50-4be3-9378-62c1870f4584-kube-api-access-z898j\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:36.061649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.061558    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1e2f6bf-fb50-4be3-9378-62c1870f4584-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:38:36.529223 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.529202    2571 generic.go:356] "Generic (PLEG): container finished" podID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerID="9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500" exitCode=0
Jul 28 17:38:36.529540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.529279    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"
Jul 28 17:38:36.529540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.529279    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerDied","Data":"9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500"}
Jul 28 17:38:36.529540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.529344    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd" event={"ID":"b1e2f6bf-fb50-4be3-9378-62c1870f4584","Type":"ContainerDied","Data":"8bcc5c7f31e0e7adeb505b5cd44846a6572bc695060c7ec78a711858c5a7632c"}
Jul 28 17:38:36.529540 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.529384    2571 scope.go:117] "RemoveContainer" containerID="fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b"
Jul 28 17:38:36.531164 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.531144    2571 generic.go:356] "Generic (PLEG): container finished" podID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerID="b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139" exitCode=0
Jul 28 17:38:36.531336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.531196    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerDied","Data":"b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139"}
Jul 28 17:38:36.531336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.531224    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"
Jul 28 17:38:36.531336 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.531234    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7" event={"ID":"89304a83-a7ee-4f86-8737-334ac79fbf66","Type":"ContainerDied","Data":"08a83bac9d1735008fe875a6baa703e68f417f83ce9b22fc9be34a838372aac5"}
Jul 28 17:38:36.540817 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.540803    2571 scope.go:117] "RemoveContainer" containerID="9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500"
Jul 28 17:38:36.547669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.547650    2571 scope.go:117] "RemoveContainer" containerID="1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043"
Jul 28 17:38:36.553781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.553765    2571 scope.go:117] "RemoveContainer" containerID="fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b"
Jul 28 17:38:36.554021 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:36.554003    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b\": container with ID starting with fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b not found: ID does not exist" containerID="fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b"
Jul 28 17:38:36.554060 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.554029    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b"} err="failed to get container status \"fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b\": rpc error: code = NotFound desc = could not find container \"fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b\": container with ID starting with fb81f2efdd4de0ef6e073e283a25a23506aba87aeab6b9e05872c9c7774dbc2b not found: ID does not exist"
Jul 28 17:38:36.554060 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.554045    2571 scope.go:117] "RemoveContainer" containerID="9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500"
Jul 28 17:38:36.554263 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:36.554236    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500\": container with ID starting with 9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500 not found: ID does not exist" containerID="9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500"
Jul 28 17:38:36.554306 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.554269    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500"} err="failed to get container status \"9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500\": rpc error: code = NotFound desc = could not find container \"9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500\": container with ID starting with 9565d0dce38809b2b61688f0858ed93cefed4e60c94403dcb8bf9ceeef35e500 not found: ID does not exist"
Jul 28 17:38:36.554306 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.554287    2571 scope.go:117] "RemoveContainer" containerID="1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043"
Jul 28 17:38:36.554531 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:36.554513    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043\": container with ID starting with 1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043 not found: ID does not exist" containerID="1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043"
Jul 28 17:38:36.554575 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.554534    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043"} err="failed to get container status \"1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043\": rpc error: code = NotFound desc = could not find container \"1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043\": container with ID starting with 1ec900ead91ec9bb208e8a6d84f5e4de01937d32389ba266ee078cbccc275043 not found: ID does not exist"
Jul 28 17:38:36.554575 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.554549    2571 scope.go:117] "RemoveContainer" containerID="e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b"
Jul 28 17:38:36.566909 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.566890    2571 scope.go:117] "RemoveContainer" containerID="b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139"
Jul 28 17:38:36.573412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.573396    2571 scope.go:117] "RemoveContainer" containerID="a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e"
Jul 28 17:38:36.577399 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.577378    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"]
Jul 28 17:38:36.579980 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.579965    2571 scope.go:117] "RemoveContainer" containerID="e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b"
Jul 28 17:38:36.580236 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:36.580218    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b\": container with ID starting with e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b not found: ID does not exist" containerID="e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b"
Jul 28 17:38:36.580320 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.580258    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b"} err="failed to get container status \"e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b\": rpc error: code = NotFound desc = could not find container \"e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b\": container with ID starting with e60fdccdd34f8e91f90f079fcf099c2a932cf2a17195f3fb76f01e04758d093b not found: ID does not exist"
Jul 28 17:38:36.580320 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.580280    2571 scope.go:117] "RemoveContainer" containerID="b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139"
Jul 28 17:38:36.580513 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:36.580496    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139\": container with ID starting with b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139 not found: ID does not exist" containerID="b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139"
Jul 28 17:38:36.580571 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.580518    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139"} err="failed to get container status \"b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139\": rpc error: code = NotFound desc = could not find container \"b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139\": container with ID starting with b1c013cddfa264da173a82482220de89ee1caaebaa6434c82b2ac4781c45f139 not found: ID does not exist"
Jul 28 17:38:36.580571 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.580531    2571 scope.go:117] "RemoveContainer" containerID="a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e"
Jul 28 17:38:36.580761 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:36.580739    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e\": container with ID starting with a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e not found: ID does not exist" containerID="a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e"
Jul 28 17:38:36.580985 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.580764    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e"} err="failed to get container status \"a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e\": rpc error: code = NotFound desc = could not find container \"a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e\": container with ID starting with a718b87ffedd28d29670339641800743340606b209edd5469ccf8f5f8e4e8f3e not found: ID does not exist"
Jul 28 17:38:36.583217 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.583197    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-svcq7"]
Jul 28 17:38:36.601994 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.601973    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"]
Jul 28 17:38:36.613545 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:36.613521    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-xszcd"]
Jul 28 17:38:37.077694 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077666    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"]
Jul 28 17:38:37.077921 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077910    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077923    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077931    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kube-rbac-proxy"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077937    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kube-rbac-proxy"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077947    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="storage-initializer"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077952    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="storage-initializer"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077958    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kube-rbac-proxy"
Jul 28 17:38:37.077964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077963    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kube-rbac-proxy"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077969    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="storage-initializer"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077974    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="storage-initializer"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077980    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.077985    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.078041    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kserve-container"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.078048    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kserve-container"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.078056    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" containerName="kube-rbac-proxy"
Jul 28 17:38:37.078168 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.078062    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" containerName="kube-rbac-proxy"
Jul 28 17:38:37.082695 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.082679    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.085510 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.085485    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:38:37.085638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.085522    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:38:37.085638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.085619    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:38:37.085770 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.085755    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:38:37.085819 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.085804    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:38:37.085881 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.085836    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:38:37.086504 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.086490    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 17:38:37.091871 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.091850    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"]
Jul 28 17:38:37.103022 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.103003    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"]
Jul 28 17:38:37.105954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.105941    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.109220 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.109198    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:38:37.117739 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.117716    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"]
Jul 28 17:38:37.168423 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.168526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f49p\" (UniqueName: \"kubernetes.io/projected/a8025c23-195a-4caa-b877-cab529c32e35-kube-api-access-9f49p\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.168526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168456    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.168526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168472    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wp8z\" (UniqueName: \"kubernetes.io/projected/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-api-access-8wp8z\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.168639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.168639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168575    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.168639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8025c23-195a-4caa-b877-cab529c32e35-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.168639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.168761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294a628a-3f6e-4e28-adcd-0f8e5b931497-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.168761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.168660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.269377 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.269474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f49p\" (UniqueName: \"kubernetes.io/projected/a8025c23-195a-4caa-b877-cab529c32e35-kube-api-access-9f49p\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.269474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269402    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.269474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wp8z\" (UniqueName: \"kubernetes.io/projected/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-api-access-8wp8z\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.269621 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:37.269504    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:38:37.269621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269548    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.269621 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:37.269567    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls podName:294a628a-3f6e-4e28-adcd-0f8e5b931497 nodeName:}" failed. No retries permitted until 2026-07-28 17:38:37.769552108 +0000 UTC m=+1702.529577714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls") pod "isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" (UID: "294a628a-3f6e-4e28-adcd-0f8e5b931497") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:38:37.269621 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.269829 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269690    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8025c23-195a-4caa-b877-cab529c32e35-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.269829 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269723    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.269829 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294a628a-3f6e-4e28-adcd-0f8e5b931497-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.269829 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.269790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.270022 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:37.269904    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 17:38:37.270022 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:38:37.269954    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls podName:a8025c23-195a-4caa-b877-cab529c32e35 nodeName:}" failed. No retries permitted until 2026-07-28 17:38:37.769940166 +0000 UTC m=+1702.529965780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls") pod "isvc-canary-promote-v2-predictor-f69c9c758-8t759" (UID: "a8025c23-195a-4caa-b877-cab529c32e35") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 28 17:38:37.270133 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.270016    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8025c23-195a-4caa-b877-cab529c32e35-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.270239 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.270135    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294a628a-3f6e-4e28-adcd-0f8e5b931497-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.270331 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.270305    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.270331 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.270316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.270406 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.270329    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.270406 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.270326    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.280086 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.280064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f49p\" (UniqueName: \"kubernetes.io/projected/a8025c23-195a-4caa-b877-cab529c32e35-kube-api-access-9f49p\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.280175 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.280119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wp8z\" (UniqueName: \"kubernetes.io/projected/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-api-access-8wp8z\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.706995 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.706972    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89304a83-a7ee-4f86-8737-334ac79fbf66" path="/var/lib/kubelet/pods/89304a83-a7ee-4f86-8737-334ac79fbf66/volumes"
Jul 28 17:38:37.707442 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.707429    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e2f6bf-fb50-4be3-9378-62c1870f4584" path="/var/lib/kubelet/pods/b1e2f6bf-fb50-4be3-9378-62c1870f4584/volumes"
Jul 28 17:38:37.774539 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.774514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.774631 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.774584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.776853 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.776829    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-8t759\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:37.776925 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.776831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-6mhnn\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:37.992509 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:37.992462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:38.015498 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.015477    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:38.130229 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.130204    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"]
Jul 28 17:38:38.133137 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:38:38.133112    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294a628a_3f6e_4e28_adcd_0f8e5b931497.slice/crio-8e7383058da643225299a872ddf636df53280c3ba3a346f8964b6096056347ea WatchSource:0}: Error finding container 8e7383058da643225299a872ddf636df53280c3ba3a346f8964b6096056347ea: Status 404 returned error can't find the container with id 8e7383058da643225299a872ddf636df53280c3ba3a346f8964b6096056347ea
Jul 28 17:38:38.152547 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.152523    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"]
Jul 28 17:38:38.165292 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:38:38.165266    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8025c23_195a_4caa_b877_cab529c32e35.slice/crio-75224a6836cea3cecfdfd026d3de88c7891c12ab3b4f62b8d4090646f0156041 WatchSource:0}: Error finding container 75224a6836cea3cecfdfd026d3de88c7891c12ab3b4f62b8d4090646f0156041: Status 404 returned error can't find the container with id 75224a6836cea3cecfdfd026d3de88c7891c12ab3b4f62b8d4090646f0156041
Jul 28 17:38:38.541982 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.541939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerStarted","Data":"213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8"}
Jul 28 17:38:38.542173 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.542019    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerStarted","Data":"75224a6836cea3cecfdfd026d3de88c7891c12ab3b4f62b8d4090646f0156041"}
Jul 28 17:38:38.543508 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.543483    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerStarted","Data":"c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9"}
Jul 28 17:38:38.543650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:38.543534    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerStarted","Data":"8e7383058da643225299a872ddf636df53280c3ba3a346f8964b6096056347ea"}
Jul 28 17:38:42.558223 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:42.558193    2571 generic.go:356] "Generic (PLEG): container finished" podID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerID="c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9" exitCode=0
Jul 28 17:38:42.558617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:42.558273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerDied","Data":"c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9"}
Jul 28 17:38:42.559877 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:42.559859    2571 generic.go:356] "Generic (PLEG): container finished" podID="a8025c23-195a-4caa-b877-cab529c32e35" containerID="213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8" exitCode=0
Jul 28 17:38:42.559960 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:42.559895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerDied","Data":"213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8"}
Jul 28 17:38:43.565613 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.565580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerStarted","Data":"c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3"}
Jul 28 17:38:43.566055 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.565641    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerStarted","Data":"1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65"}
Jul 28 17:38:43.566055 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.565891    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:43.567711 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.567682    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerStarted","Data":"72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd"}
Jul 28 17:38:43.567832 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.567748    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerStarted","Data":"0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7"}
Jul 28 17:38:43.567968 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.567950    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:43.568013 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.568002    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:43.569083 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.569060    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:38:43.590242 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.590197    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podStartSLOduration=2.825981554 podStartE2EDuration="6.590186469s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.764204915s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:43.589686923 +0000 UTC m=+1708.349712551" watchObservedRunningTime="2026-07-28 17:38:43.590186469 +0000 UTC m=+1708.350212097"
Jul 28 17:38:43.610715 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:43.610676    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podStartSLOduration=2.898356884 podStartE2EDuration="6.610664533s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.712307649s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:43.609920632 +0000 UTC m=+1708.369946293" watchObservedRunningTime="2026-07-28 17:38:43.610664533 +0000 UTC m=+1708.370690215"
Jul 28 17:38:44.571043 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:44.571004    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:38:44.571440 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:44.571273    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:44.572367 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:44.572341    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:45.573987 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:45.573948    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:49.574931 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:49.574893    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:38:49.575440 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:49.575413    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:38:50.578939 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:50.578912    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:38:50.579489 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:50.579460    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:59.575595 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:38:59.575555    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:39:00.579614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:00.579579    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:39:09.576038 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:09.576004    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:39:10.580363 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:10.580330    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:39:19.576018 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:19.575971    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:39:20.579982 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:20.579943    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:39:29.575614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:29.575576    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:39:30.579754 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:30.579715    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:39:39.576288 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:39.576230    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:39:40.579790 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:40.579737    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:39:49.576169 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:49.576140    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:39:50.580394 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:50.580368    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:39:57.160582 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:57.160554    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"]
Jul 28 17:39:57.160966 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:57.160775    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" containerID="cri-o://0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7" gracePeriod=30
Jul 28 17:39:57.160966 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:57.160803    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kube-rbac-proxy" containerID="cri-o://72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd" gracePeriod=30
Jul 28 17:39:57.828406 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:57.828378    2571 generic.go:356] "Generic (PLEG): container finished" podID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerID="72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd" exitCode=2
Jul 28 17:39:57.828555 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:57.828450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerDied","Data":"72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd"}
Jul 28 17:39:59.571451 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:59.571413    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 28 17:39:59.575657 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:39:59.575633    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:40:00.505256 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.505222    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:40:00.628524 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628452    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wp8z\" (UniqueName: \"kubernetes.io/projected/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-api-access-8wp8z\") pod \"294a628a-3f6e-4e28-adcd-0f8e5b931497\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") "
Jul 28 17:40:00.628524 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628504    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-rbac-proxy-sar-config\") pod \"294a628a-3f6e-4e28-adcd-0f8e5b931497\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") "
Jul 28 17:40:00.628524 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628523    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls\") pod \"294a628a-3f6e-4e28-adcd-0f8e5b931497\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") "
Jul 28 17:40:00.628937 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628541    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-cabundle-cert\") pod \"294a628a-3f6e-4e28-adcd-0f8e5b931497\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") "
Jul 28 17:40:00.628937 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628581    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294a628a-3f6e-4e28-adcd-0f8e5b931497-kserve-provision-location\") pod \"294a628a-3f6e-4e28-adcd-0f8e5b931497\" (UID: \"294a628a-3f6e-4e28-adcd-0f8e5b931497\") "
Jul 28 17:40:00.629017 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628940    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "294a628a-3f6e-4e28-adcd-0f8e5b931497" (UID: "294a628a-3f6e-4e28-adcd-0f8e5b931497"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:00.629017 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628955    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "294a628a-3f6e-4e28-adcd-0f8e5b931497" (UID: "294a628a-3f6e-4e28-adcd-0f8e5b931497"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:00.629017 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.628960    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/294a628a-3f6e-4e28-adcd-0f8e5b931497-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "294a628a-3f6e-4e28-adcd-0f8e5b931497" (UID: "294a628a-3f6e-4e28-adcd-0f8e5b931497"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:00.630526 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.630506    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "294a628a-3f6e-4e28-adcd-0f8e5b931497" (UID: "294a628a-3f6e-4e28-adcd-0f8e5b931497"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:00.630597 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.630550    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-api-access-8wp8z" (OuterVolumeSpecName: "kube-api-access-8wp8z") pod "294a628a-3f6e-4e28-adcd-0f8e5b931497" (UID: "294a628a-3f6e-4e28-adcd-0f8e5b931497"). InnerVolumeSpecName "kube-api-access-8wp8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:00.729781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.729750    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wp8z\" (UniqueName: \"kubernetes.io/projected/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-api-access-8wp8z\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:40:00.729781 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.729779    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:40:00.729907 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.729793    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/294a628a-3f6e-4e28-adcd-0f8e5b931497-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:40:00.729907 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.729807    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/294a628a-3f6e-4e28-adcd-0f8e5b931497-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:40:00.729907 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.729818    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/294a628a-3f6e-4e28-adcd-0f8e5b931497-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:40:00.839607 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.839579    2571 generic.go:356] "Generic (PLEG): container finished" podID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerID="0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7" exitCode=0
Jul 28 17:40:00.839704 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.839637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerDied","Data":"0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7"}
Jul 28 17:40:00.839704 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.839691    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn" event={"ID":"294a628a-3f6e-4e28-adcd-0f8e5b931497","Type":"ContainerDied","Data":"8e7383058da643225299a872ddf636df53280c3ba3a346f8964b6096056347ea"}
Jul 28 17:40:00.839812 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.839641    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"
Jul 28 17:40:00.839812 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.839727    2571 scope.go:117] "RemoveContainer" containerID="72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd"
Jul 28 17:40:00.847230 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.847133    2571 scope.go:117] "RemoveContainer" containerID="0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7"
Jul 28 17:40:00.855057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.855040    2571 scope.go:117] "RemoveContainer" containerID="c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9"
Jul 28 17:40:00.861633 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.861613    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"]
Jul 28 17:40:00.862450 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.862434    2571 scope.go:117] "RemoveContainer" containerID="72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd"
Jul 28 17:40:00.862676 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:40:00.862661    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd\": container with ID starting with 72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd not found: ID does not exist" containerID="72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd"
Jul 28 17:40:00.862744 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.862683    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd"} err="failed to get container status \"72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd\": rpc error: code = NotFound desc = could not find container \"72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd\": container with ID starting with 72b59b666ad037bd8e71abbe09abc89839f2d10e6dd1fd39f0d758ea1beb96bd not found: ID does not exist"
Jul 28 17:40:00.862744 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.862703    2571 scope.go:117] "RemoveContainer" containerID="0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7"
Jul 28 17:40:00.862936 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:40:00.862922    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7\": container with ID starting with 0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7 not found: ID does not exist" containerID="0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7"
Jul 28 17:40:00.862981 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.862939    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7"} err="failed to get container status \"0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7\": rpc error: code = NotFound desc = could not find container \"0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7\": container with ID starting with 0c4c5e795b1ed36297f80f4ce81c53e78dd28cf35020d480f47061df0aab05d7 not found: ID does not exist"
Jul 28 17:40:00.862981 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.862950    2571 scope.go:117] "RemoveContainer" containerID="c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9"
Jul 28 17:40:00.863152 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:40:00.863134    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9\": container with ID starting with c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9 not found: ID does not exist" containerID="c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9"
Jul 28 17:40:00.863190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.863159    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9"} err="failed to get container status \"c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9\": rpc error: code = NotFound desc = could not find container \"c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9\": container with ID starting with c111fda78e9d01c46c8c374044c3948dc72a476b593eee4a94900c594b33c4f9 not found: ID does not exist"
Jul 28 17:40:00.864969 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:00.864952    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-6mhnn"]
Jul 28 17:40:01.707386 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:40:01.707361    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" path="/var/lib/kubelet/pods/294a628a-3f6e-4e28-adcd-0f8e5b931497/volumes"
Jul 28 17:41:37.460966 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:37.460936    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"]
Jul 28 17:41:37.461488 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:37.461140    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" containerID="cri-o://1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65" gracePeriod=30
Jul 28 17:41:37.461488 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:37.461203    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kube-rbac-proxy" containerID="cri-o://c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3" gracePeriod=30
Jul 28 17:41:38.163677 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:38.163649    2571 generic.go:356] "Generic (PLEG): container finished" podID="a8025c23-195a-4caa-b877-cab529c32e35" containerID="c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3" exitCode=2
Jul 28 17:41:38.163850 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:38.163703    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerDied","Data":"c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3"}
Jul 28 17:41:40.575166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:40.575120    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 17:41:40.579519 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:40.579493    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:41:40.995422 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:40.995402    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:41:41.078497 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078475    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f49p\" (UniqueName: \"kubernetes.io/projected/a8025c23-195a-4caa-b877-cab529c32e35-kube-api-access-9f49p\") pod \"a8025c23-195a-4caa-b877-cab529c32e35\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") "
Jul 28 17:41:41.078613 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078511    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8025c23-195a-4caa-b877-cab529c32e35-kserve-provision-location\") pod \"a8025c23-195a-4caa-b877-cab529c32e35\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") "
Jul 28 17:41:41.078613 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078536    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls\") pod \"a8025c23-195a-4caa-b877-cab529c32e35\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") "
Jul 28 17:41:41.078691 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078656    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-kube-rbac-proxy-sar-config\") pod \"a8025c23-195a-4caa-b877-cab529c32e35\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") "
Jul 28 17:41:41.078738 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078686    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-cabundle-cert\") pod \"a8025c23-195a-4caa-b877-cab529c32e35\" (UID: \"a8025c23-195a-4caa-b877-cab529c32e35\") "
Jul 28 17:41:41.078825 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078795    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8025c23-195a-4caa-b877-cab529c32e35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8025c23-195a-4caa-b877-cab529c32e35" (UID: "a8025c23-195a-4caa-b877-cab529c32e35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:41:41.078979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078962    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8025c23-195a-4caa-b877-cab529c32e35-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:41:41.079048 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.078983    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8025c23-195a-4caa-b877-cab529c32e35" (UID: "a8025c23-195a-4caa-b877-cab529c32e35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:41.079128 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.079104    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8025c23-195a-4caa-b877-cab529c32e35" (UID: "a8025c23-195a-4caa-b877-cab529c32e35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:41.080426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.080403    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8025c23-195a-4caa-b877-cab529c32e35" (UID: "a8025c23-195a-4caa-b877-cab529c32e35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:41.080525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.080410    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8025c23-195a-4caa-b877-cab529c32e35-kube-api-access-9f49p" (OuterVolumeSpecName: "kube-api-access-9f49p") pod "a8025c23-195a-4caa-b877-cab529c32e35" (UID: "a8025c23-195a-4caa-b877-cab529c32e35"). InnerVolumeSpecName "kube-api-access-9f49p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:41.175471 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.175446    2571 generic.go:356] "Generic (PLEG): container finished" podID="a8025c23-195a-4caa-b877-cab529c32e35" containerID="1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65" exitCode=0
Jul 28 17:41:41.175566 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.175495    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"
Jul 28 17:41:41.175566 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.175503    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerDied","Data":"1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65"}
Jul 28 17:41:41.175566 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.175549    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759" event={"ID":"a8025c23-195a-4caa-b877-cab529c32e35","Type":"ContainerDied","Data":"75224a6836cea3cecfdfd026d3de88c7891c12ab3b4f62b8d4090646f0156041"}
Jul 28 17:41:41.175669 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.175573    2571 scope.go:117] "RemoveContainer" containerID="c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3"
Jul 28 17:41:41.179438 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.179419    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:41:41.179525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.179443    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8025c23-195a-4caa-b877-cab529c32e35-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:41:41.179525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.179457    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f49p\" (UniqueName: \"kubernetes.io/projected/a8025c23-195a-4caa-b877-cab529c32e35-kube-api-access-9f49p\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:41:41.179525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.179469    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8025c23-195a-4caa-b877-cab529c32e35-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:41:41.185011 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.184992    2571 scope.go:117] "RemoveContainer" containerID="1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65"
Jul 28 17:41:41.191563 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.191547    2571 scope.go:117] "RemoveContainer" containerID="213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8"
Jul 28 17:41:41.197771 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.197757    2571 scope.go:117] "RemoveContainer" containerID="c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3"
Jul 28 17:41:41.198009 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:41:41.197991    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3\": container with ID starting with c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3 not found: ID does not exist" containerID="c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3"
Jul 28 17:41:41.198078 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.198019    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3"} err="failed to get container status \"c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3\": rpc error: code = NotFound desc = could not find container \"c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3\": container with ID starting with c57265efc4cfd35076d7c3bdd30bbbfa104a2ac3b1c2f74c8b3882083a0642a3 not found: ID does not exist"
Jul 28 17:41:41.198078 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.198043    2571 scope.go:117] "RemoveContainer" containerID="1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65"
Jul 28 17:41:41.198382 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:41:41.198355    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65\": container with ID starting with 1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65 not found: ID does not exist" containerID="1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65"
Jul 28 17:41:41.198485 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.198390    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65"} err="failed to get container status \"1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65\": rpc error: code = NotFound desc = could not find container \"1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65\": container with ID starting with 1f5edb53a0bbbe08e201c2685b75361504b1f8b978f31a85ec6d3f9b015dfd65 not found: ID does not exist"
Jul 28 17:41:41.198485 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.198410    2571 scope.go:117] "RemoveContainer" containerID="213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8"
Jul 28 17:41:41.198699 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:41:41.198677    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8\": container with ID starting with 213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8 not found: ID does not exist" containerID="213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8"
Jul 28 17:41:41.198837 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.198809    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8"} err="failed to get container status \"213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8\": rpc error: code = NotFound desc = could not find container \"213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8\": container with ID starting with 213eb28338d4181f7fda02a86ac3e8a8e1ab51daf9f5e26ef86e639f68872dc8 not found: ID does not exist"
Jul 28 17:41:41.200242 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.200221    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"]
Jul 28 17:41:41.204405 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.204387    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-8t759"]
Jul 28 17:41:41.706947 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:41.706913    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8025c23-195a-4caa-b877-cab529c32e35" path="/var/lib/kubelet/pods/a8025c23-195a-4caa-b877-cab529c32e35/volumes"
Jul 28 17:41:42.638198 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638169    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"]
Jul 28 17:41:42.638485 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638470    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="storage-initializer"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638488    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="storage-initializer"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638508    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="storage-initializer"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638516    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="storage-initializer"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638526    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kube-rbac-proxy"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638536    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kube-rbac-proxy"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638548    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638556    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638568    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container"
Jul 28 17:41:42.638578 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638576    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container"
Jul 28 17:41:42.638997 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638590    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kube-rbac-proxy"
Jul 28 17:41:42.638997 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638601    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kube-rbac-proxy"
Jul 28 17:41:42.638997 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638663    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kube-rbac-proxy"
Jul 28 17:41:42.638997 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638676    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kube-rbac-proxy"
Jul 28 17:41:42.638997 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638685    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="294a628a-3f6e-4e28-adcd-0f8e5b931497" containerName="kserve-container"
Jul 28 17:41:42.638997 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.638694    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8025c23-195a-4caa-b877-cab529c32e35" containerName="kserve-container"
Jul 28 17:41:42.643186 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.643167    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.645633 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.645613    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 17:41:42.645726 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.645705    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:41:42.646077 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.646059    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:41:42.646840 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.646822    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:41:42.646949 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.646839    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:41:42.646949 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.646861    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:41:42.646949 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.646884    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:41:42.652488 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.652467    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"]
Jul 28 17:41:42.789683 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.789661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.789992 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.789698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.789992 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.789726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzkch\" (UniqueName: \"kubernetes.io/projected/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-api-access-vzkch\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.789992 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.789867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.789992 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.789900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.890457 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.890405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.890457 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.890435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzkch\" (UniqueName: \"kubernetes.io/projected/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-api-access-vzkch\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.890606 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.890471    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.890606 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.890496    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.890606 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.890596    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.890875 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.890856    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.891077 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.891061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.891177 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.891159    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.892908 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.892892    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.898344 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.898322    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzkch\" (UniqueName: \"kubernetes.io/projected/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-api-access-vzkch\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:42.953138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:42.953117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:43.079782 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:43.079728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"]
Jul 28 17:41:43.082046 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:41:43.082021    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6ac9f6_afb4_4cf7_9191_a5e3406d197d.slice/crio-a5df8177c73e0dc86b98d3320ff819ec8df8f4831f94bf51b82a62a09c55f6cd WatchSource:0}: Error finding container a5df8177c73e0dc86b98d3320ff819ec8df8f4831f94bf51b82a62a09c55f6cd: Status 404 returned error can't find the container with id a5df8177c73e0dc86b98d3320ff819ec8df8f4831f94bf51b82a62a09c55f6cd
Jul 28 17:41:43.183454 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:43.183392    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerStarted","Data":"8d094816b092d0e0535e07a4ee058916f495fe7c50c193cd717b0ef022c57b00"}
Jul 28 17:41:43.183566 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:43.183452    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerStarted","Data":"a5df8177c73e0dc86b98d3320ff819ec8df8f4831f94bf51b82a62a09c55f6cd"}
Jul 28 17:41:47.197038 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:47.197012    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerID="8d094816b092d0e0535e07a4ee058916f495fe7c50c193cd717b0ef022c57b00" exitCode=0
Jul 28 17:41:47.197373 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:47.197087    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerDied","Data":"8d094816b092d0e0535e07a4ee058916f495fe7c50c193cd717b0ef022c57b00"}
Jul 28 17:41:48.202357 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:48.202323    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerStarted","Data":"219324c5612e150cb4f2eb679010caa745660bd4bc8f65a65efaf5d8b0ba299c"}
Jul 28 17:41:48.202707 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:48.202380    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerStarted","Data":"3fb4c6bef565b0d0f2b2a32028d5192bfd0ce27839c0d4cebce96037ba7c5338"}
Jul 28 17:41:48.202707 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:48.202560    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:48.229071 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:48.229026    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podStartSLOduration=2.393138096 podStartE2EDuration="6.229010946s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83587285s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:48.227518518 +0000 UTC m=+1892.987544180" watchObservedRunningTime="2026-07-28 17:41:48.229010946 +0000 UTC m=+1892.989036637"
Jul 28 17:41:49.205912 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:49.205881    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:49.207014 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:49.206992    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:41:50.208740 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:50.208706    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:41:55.213326 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:55.213296    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:41:55.213782 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:41:55.213755    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:42:05.214451 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:42:05.214410    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:42:15.213995 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:42:15.213959    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:42:25.214441 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:42:25.214404    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:42:35.214443 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:42:35.214403    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:42:45.214143 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:42:45.214108    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:42:55.214934 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:42:55.214902    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:43:02.643466 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.643430    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"]
Jul 28 17:43:02.643904 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.643703    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" containerID="cri-o://3fb4c6bef565b0d0f2b2a32028d5192bfd0ce27839c0d4cebce96037ba7c5338" gracePeriod=30
Jul 28 17:43:02.643904 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.643718    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kube-rbac-proxy" containerID="cri-o://219324c5612e150cb4f2eb679010caa745660bd4bc8f65a65efaf5d8b0ba299c" gracePeriod=30
Jul 28 17:43:02.691024 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.690995    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"]
Jul 28 17:43:02.693861 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.693846    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.697153 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.697133    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:43:02.708359 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.708340    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"]
Jul 28 17:43:02.716351 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.716334    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxcd\" (UniqueName: \"kubernetes.io/projected/4c470cff-96da-4dd1-b464-db32219cc308-kube-api-access-zcxcd\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.716445 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.716374    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.716445 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.716397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c470cff-96da-4dd1-b464-db32219cc308-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.716523 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.716457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c470cff-96da-4dd1-b464-db32219cc308-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.716523 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.716488    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.817652 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.817630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxcd\" (UniqueName: \"kubernetes.io/projected/4c470cff-96da-4dd1-b464-db32219cc308-kube-api-access-zcxcd\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.817761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.817666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.817761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.817688    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c470cff-96da-4dd1-b464-db32219cc308-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.817761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.817713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c470cff-96da-4dd1-b464-db32219cc308-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.817761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.817729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.818096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.818076    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c470cff-96da-4dd1-b464-db32219cc308-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.818302 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.818283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.818380 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.818366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.819948 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.819928    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c470cff-96da-4dd1-b464-db32219cc308-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:02.827628 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:02.827607    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxcd\" (UniqueName: \"kubernetes.io/projected/4c470cff-96da-4dd1-b464-db32219cc308-kube-api-access-zcxcd\") pod \"isvc-canary-rollback-predictor-b4645bb5c-2f4jv\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:03.003104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:03.003045    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:03.128053 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:03.128022    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"]
Jul 28 17:43:03.129538 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:43:03.129514    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c470cff_96da_4dd1_b464_db32219cc308.slice/crio-36a33d4032deebe199c1cc2ea2b4c6935871e281de6220c5ee06a4347d6dcff8 WatchSource:0}: Error finding container 36a33d4032deebe199c1cc2ea2b4c6935871e281de6220c5ee06a4347d6dcff8: Status 404 returned error can't find the container with id 36a33d4032deebe199c1cc2ea2b4c6935871e281de6220c5ee06a4347d6dcff8
Jul 28 17:43:03.458936 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:03.458899    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerStarted","Data":"0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277"}
Jul 28 17:43:03.459129 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:03.458999    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerStarted","Data":"36a33d4032deebe199c1cc2ea2b4c6935871e281de6220c5ee06a4347d6dcff8"}
Jul 28 17:43:03.461030 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:03.461004    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerID="219324c5612e150cb4f2eb679010caa745660bd4bc8f65a65efaf5d8b0ba299c" exitCode=2
Jul 28 17:43:03.461154 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:03.461038    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerDied","Data":"219324c5612e150cb4f2eb679010caa745660bd4bc8f65a65efaf5d8b0ba299c"}
Jul 28 17:43:05.209145 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:05.209108    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 17:43:05.214523 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:05.214489    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:43:06.473923 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.473893    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerID="3fb4c6bef565b0d0f2b2a32028d5192bfd0ce27839c0d4cebce96037ba7c5338" exitCode=0
Jul 28 17:43:06.474258 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.473939    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerDied","Data":"3fb4c6bef565b0d0f2b2a32028d5192bfd0ce27839c0d4cebce96037ba7c5338"}
Jul 28 17:43:06.591419 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.591396    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:43:06.644217 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644196    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-cabundle-cert\") pod \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") "
Jul 28 17:43:06.644332 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644233    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-rbac-proxy-sar-config\") pod \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") "
Jul 28 17:43:06.644332 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644284    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kserve-provision-location\") pod \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") "
Jul 28 17:43:06.644444 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644339    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-proxy-tls\") pod \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") "
Jul 28 17:43:06.644444 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644363    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzkch\" (UniqueName: \"kubernetes.io/projected/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-api-access-vzkch\") pod \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\" (UID: \"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d\") "
Jul 28 17:43:06.644594 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644567    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" (UID: "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:06.644657 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644623    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" (UID: "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:06.644657 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.644647    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" (UID: "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:06.646271 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.646236    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" (UID: "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:06.646353 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.646335    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-api-access-vzkch" (OuterVolumeSpecName: "kube-api-access-vzkch") pod "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" (UID: "ed6ac9f6-afb4-4cf7-9191-a5e3406d197d"). InnerVolumeSpecName "kube-api-access-vzkch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:06.745035 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.745003    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:43:06.745035 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.745034    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzkch\" (UniqueName: \"kubernetes.io/projected/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-api-access-vzkch\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:43:06.745035 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.745044    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:43:06.745185 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.745052    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:43:06.745185 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:06.745061    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:43:07.479401 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.479375    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"
Jul 28 17:43:07.479785 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.479376    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4" event={"ID":"ed6ac9f6-afb4-4cf7-9191-a5e3406d197d","Type":"ContainerDied","Data":"a5df8177c73e0dc86b98d3320ff819ec8df8f4831f94bf51b82a62a09c55f6cd"}
Jul 28 17:43:07.479785 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.479534    2571 scope.go:117] "RemoveContainer" containerID="219324c5612e150cb4f2eb679010caa745660bd4bc8f65a65efaf5d8b0ba299c"
Jul 28 17:43:07.481030 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.481009    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c470cff-96da-4dd1-b464-db32219cc308" containerID="0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277" exitCode=0
Jul 28 17:43:07.481152 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.481038    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerDied","Data":"0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277"}
Jul 28 17:43:07.487860 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.487839    2571 scope.go:117] "RemoveContainer" containerID="3fb4c6bef565b0d0f2b2a32028d5192bfd0ce27839c0d4cebce96037ba7c5338"
Jul 28 17:43:07.494706 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.494689    2571 scope.go:117] "RemoveContainer" containerID="8d094816b092d0e0535e07a4ee058916f495fe7c50c193cd717b0ef022c57b00"
Jul 28 17:43:07.552076 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.552053    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"]
Jul 28 17:43:07.553507 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.553489    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-trdq4"]
Jul 28 17:43:07.707649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:07.707622    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" path="/var/lib/kubelet/pods/ed6ac9f6-afb4-4cf7-9191-a5e3406d197d/volumes"
Jul 28 17:43:08.486666 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:08.486600    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerStarted","Data":"0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8"}
Jul 28 17:43:08.486989 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:08.486664    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerStarted","Data":"000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2"}
Jul 28 17:43:08.486989 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:08.486859    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:08.512516 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:08.512473    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podStartSLOduration=2.680651551 podStartE2EDuration="6.512459848s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.831808297s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:08.510984123 +0000 UTC m=+1973.271009766" watchObservedRunningTime="2026-07-28 17:43:08.512459848 +0000 UTC m=+1973.272485475"
Jul 28 17:43:09.489959 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:09.489931    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:09.491225 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:09.491194    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:43:10.492719 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:10.492686    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:43:15.497242 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:15.497217    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:43:15.497725 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:15.497699    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:43:25.497638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:25.497602    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:43:35.498549 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:35.498513    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:43:45.498092 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:45.498056    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:43:55.498339 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:43:55.498297    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:44:05.498475 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:05.498436    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:44:15.498140 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:15.498111    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:44:22.791269 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:22.791227    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"]
Jul 28 17:44:22.791649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:22.791465    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" containerID="cri-o://000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2" gracePeriod=30
Jul 28 17:44:22.791649 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:22.791504    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kube-rbac-proxy" containerID="cri-o://0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8" gracePeriod=30
Jul 28 17:44:23.735376 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:23.735343    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c470cff-96da-4dd1-b464-db32219cc308" containerID="0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8" exitCode=2
Jul 28 17:44:23.735533 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:23.735414    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerDied","Data":"0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8"}
Jul 28 17:44:25.492970 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:25.492922    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:44:25.497638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:25.497615    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:44:26.431226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.431204    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:44:26.557352 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557294    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c470cff-96da-4dd1-b464-db32219cc308-proxy-tls\") pod \"4c470cff-96da-4dd1-b464-db32219cc308\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") "
Jul 28 17:44:26.557352 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557342    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-cabundle-cert\") pod \"4c470cff-96da-4dd1-b464-db32219cc308\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") "
Jul 28 17:44:26.557776 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557365    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c470cff-96da-4dd1-b464-db32219cc308-kserve-provision-location\") pod \"4c470cff-96da-4dd1-b464-db32219cc308\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") "
Jul 28 17:44:26.557776 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557385    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcxcd\" (UniqueName: \"kubernetes.io/projected/4c470cff-96da-4dd1-b464-db32219cc308-kube-api-access-zcxcd\") pod \"4c470cff-96da-4dd1-b464-db32219cc308\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") "
Jul 28 17:44:26.557776 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557412    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-kube-rbac-proxy-sar-config\") pod \"4c470cff-96da-4dd1-b464-db32219cc308\" (UID: \"4c470cff-96da-4dd1-b464-db32219cc308\") "
Jul 28 17:44:26.557776 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557610    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c470cff-96da-4dd1-b464-db32219cc308-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c470cff-96da-4dd1-b464-db32219cc308" (UID: "4c470cff-96da-4dd1-b464-db32219cc308"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:26.557776 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557747    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c470cff-96da-4dd1-b464-db32219cc308" (UID: "4c470cff-96da-4dd1-b464-db32219cc308"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:26.557947 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.557796    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c470cff-96da-4dd1-b464-db32219cc308" (UID: "4c470cff-96da-4dd1-b464-db32219cc308"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:26.559297 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.559270    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c470cff-96da-4dd1-b464-db32219cc308-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c470cff-96da-4dd1-b464-db32219cc308" (UID: "4c470cff-96da-4dd1-b464-db32219cc308"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:26.559400 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.559319    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c470cff-96da-4dd1-b464-db32219cc308-kube-api-access-zcxcd" (OuterVolumeSpecName: "kube-api-access-zcxcd") pod "4c470cff-96da-4dd1-b464-db32219cc308" (UID: "4c470cff-96da-4dd1-b464-db32219cc308"). InnerVolumeSpecName "kube-api-access-zcxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:26.658031 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.658009    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxcd\" (UniqueName: \"kubernetes.io/projected/4c470cff-96da-4dd1-b464-db32219cc308-kube-api-access-zcxcd\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:44:26.658031 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.658029    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:44:26.658150 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.658039    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c470cff-96da-4dd1-b464-db32219cc308-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:44:26.658150 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.658048    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c470cff-96da-4dd1-b464-db32219cc308-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:44:26.658150 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.658056    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c470cff-96da-4dd1-b464-db32219cc308-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:44:26.747990 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.747966    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c470cff-96da-4dd1-b464-db32219cc308" containerID="000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2" exitCode=0
Jul 28 17:44:26.748096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.747997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerDied","Data":"000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2"}
Jul 28 17:44:26.748096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.748015    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"
Jul 28 17:44:26.748096 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.748064    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv" event={"ID":"4c470cff-96da-4dd1-b464-db32219cc308","Type":"ContainerDied","Data":"36a33d4032deebe199c1cc2ea2b4c6935871e281de6220c5ee06a4347d6dcff8"}
Jul 28 17:44:26.748279 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.748103    2571 scope.go:117] "RemoveContainer" containerID="0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8"
Jul 28 17:44:26.755824 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.755805    2571 scope.go:117] "RemoveContainer" containerID="000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2"
Jul 28 17:44:26.762482 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.762468    2571 scope.go:117] "RemoveContainer" containerID="0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277"
Jul 28 17:44:26.769052 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.769037    2571 scope.go:117] "RemoveContainer" containerID="0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8"
Jul 28 17:44:26.769349 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:44:26.769331    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8\": container with ID starting with 0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8 not found: ID does not exist" containerID="0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8"
Jul 28 17:44:26.769420 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.769361    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8"} err="failed to get container status \"0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8\": rpc error: code = NotFound desc = could not find container \"0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8\": container with ID starting with 0bc1251bfb8b28877bf8316f8a849b09c8bbd3f31017c63b6a0f41fff1fc2cf8 not found: ID does not exist"
Jul 28 17:44:26.769420 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.769383    2571 scope.go:117] "RemoveContainer" containerID="000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2"
Jul 28 17:44:26.769636 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:44:26.769620    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2\": container with ID starting with 000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2 not found: ID does not exist" containerID="000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2"
Jul 28 17:44:26.769676 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.769642    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2"} err="failed to get container status \"000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2\": rpc error: code = NotFound desc = could not find container \"000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2\": container with ID starting with 000fcf5627e665b2d8211b100f38a32e1f056805692cb7fdfb6cfb2d1389f9b2 not found: ID does not exist"
Jul 28 17:44:26.769676 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.769657    2571 scope.go:117] "RemoveContainer" containerID="0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277"
Jul 28 17:44:26.769902 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:44:26.769880    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277\": container with ID starting with 0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277 not found: ID does not exist" containerID="0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277"
Jul 28 17:44:26.769959 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.769911    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277"} err="failed to get container status \"0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277\": rpc error: code = NotFound desc = could not find container \"0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277\": container with ID starting with 0acdeb7a61db140bea0626f384e889bf10c6d7ecfc6c4006d943994e532ba277 not found: ID does not exist"
Jul 28 17:44:26.770241 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.770227    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"]
Jul 28 17:44:26.774531 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:26.774512    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-2f4jv"]
Jul 28 17:44:27.706969 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:27.706944    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c470cff-96da-4dd1-b464-db32219cc308" path="/var/lib/kubelet/pods/4c470cff-96da-4dd1-b464-db32219cc308/volumes"
Jul 28 17:44:28.045747 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.045673    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"]
Jul 28 17:44:28.045980 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.045962    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.045983    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.045996    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="storage-initializer"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046004    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="storage-initializer"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046016    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kube-rbac-proxy"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046025    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kube-rbac-proxy"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046042    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="storage-initializer"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046050    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="storage-initializer"
Jul 28 17:44:28.046065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046064    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046072    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046090    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kube-rbac-proxy"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046098    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kube-rbac-proxy"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046156    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kube-rbac-proxy"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046168    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kserve-container"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046177    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c470cff-96da-4dd1-b464-db32219cc308" containerName="kube-rbac-proxy"
Jul 28 17:44:28.046484 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.046186    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6ac9f6-afb4-4cf7-9191-a5e3406d197d" containerName="kserve-container"
Jul 28 17:44:28.050757 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.050737    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.053703 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.053674    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 17:44:28.053703 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.053676    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:44:28.053861 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.053728    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:44:28.053861 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.053700    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:44:28.053967 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.053945    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:44:28.054819 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.054801    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:44:28.054887 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.054872    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:44:28.061334 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.061315    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"]
Jul 28 17:44:28.170075 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.170048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jpr\" (UniqueName: \"kubernetes.io/projected/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-api-access-m9jpr\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.170165 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.170090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.170165 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.170113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.170236 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.170169    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.170236 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.170212    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.271331 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.271298    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jpr\" (UniqueName: \"kubernetes.io/projected/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-api-access-m9jpr\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.271412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.271348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.271412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.271371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.271412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.271405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.271537 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.271445    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.271762 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.271743    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.272031 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.272013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.272093 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.272052    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.273729 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.273708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.280312 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.280293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jpr\" (UniqueName: \"kubernetes.io/projected/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-api-access-m9jpr\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-7sq78\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.360307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.360256    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:28.483718 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.483690    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"]
Jul 28 17:44:28.486630 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:44:28.486602    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e36b9da_7ec4_4a2c_b8cd_3242dbc66eb3.slice/crio-122621b3e841d633c79203e57d16448c540d445ed04eb661a1303f8d4e54c86b WatchSource:0}: Error finding container 122621b3e841d633c79203e57d16448c540d445ed04eb661a1303f8d4e54c86b: Status 404 returned error can't find the container with id 122621b3e841d633c79203e57d16448c540d445ed04eb661a1303f8d4e54c86b
Jul 28 17:44:28.757397 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.757368    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerStarted","Data":"8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a"}
Jul 28 17:44:28.757724 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:28.757423    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerStarted","Data":"122621b3e841d633c79203e57d16448c540d445ed04eb661a1303f8d4e54c86b"}
Jul 28 17:44:32.772321 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:32.772290    2571 generic.go:356] "Generic (PLEG): container finished" podID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerID="8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a" exitCode=0
Jul 28 17:44:32.772663 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:32.772336    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerDied","Data":"8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a"}
Jul 28 17:44:33.777178 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:33.777145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerStarted","Data":"f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7"}
Jul 28 17:44:33.777581 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:33.777197    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerStarted","Data":"2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6"}
Jul 28 17:44:33.777581 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:33.777467    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:33.777653 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:33.777611    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:33.778746 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:33.778720    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:44:33.801435 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:33.801396    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podStartSLOduration=2.128196167 podStartE2EDuration="5.801385002s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.673188835s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:33.800537316 +0000 UTC m=+2058.560562979" watchObservedRunningTime="2026-07-28 17:44:33.801385002 +0000 UTC m=+2058.561410630"
Jul 28 17:44:34.780436 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:34.780404    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:44:35.782877 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:35.782838    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:44:40.787681 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:40.787612    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:44:40.788157 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:40.788130    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:44:50.788754 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:44:50.788718    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:45:00.788961 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:00.788923    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:45:10.788113 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:10.788076    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:45:20.789040 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:20.789005    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:45:30.788906 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:30.788872    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:45:40.789275 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:40.789232    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:45:58.070095 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:58.070060    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"]
Jul 28 17:45:58.070591 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:58.070285    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" containerID="cri-o://2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6" gracePeriod=30
Jul 28 17:45:58.070591 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:58.070319    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kube-rbac-proxy" containerID="cri-o://f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7" gracePeriod=30
Jul 28 17:45:59.054425 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:59.054391    2571 generic.go:356] "Generic (PLEG): container finished" podID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerID="f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7" exitCode=2
Jul 28 17:45:59.054589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:45:59.054437    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerDied","Data":"f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7"}
Jul 28 17:46:00.783164 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:00.783123    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 17:46:00.789061 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:00.789036    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:46:01.504896 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.504874    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:46:01.577944 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.577922    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9jpr\" (UniqueName: \"kubernetes.io/projected/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-api-access-m9jpr\") pod \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") "
Jul 28 17:46:01.578066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.577968    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kserve-provision-location\") pod \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") "
Jul 28 17:46:01.578066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.577989    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-rbac-proxy-sar-config\") pod \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") "
Jul 28 17:46:01.578066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.578031    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-proxy-tls\") pod \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") "
Jul 28 17:46:01.578066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.578044    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-cabundle-cert\") pod \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\" (UID: \"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3\") "
Jul 28 17:46:01.578314 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.578275    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" (UID: "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:01.578434 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.578345    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" (UID: "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:01.578543 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.578494    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" (UID: "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:01.579914 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.579888    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-api-access-m9jpr" (OuterVolumeSpecName: "kube-api-access-m9jpr") pod "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" (UID: "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3"). InnerVolumeSpecName "kube-api-access-m9jpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:01.580003 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.579949    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" (UID: "2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:01.678457 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.678435    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:46:01.678457 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.678455    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:46:01.678574 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.678465    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:46:01.678574 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.678474    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:46:01.678574 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:01.678482    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9jpr\" (UniqueName: \"kubernetes.io/projected/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3-kube-api-access-m9jpr\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:46:02.068296 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.068209    2571 generic.go:356] "Generic (PLEG): container finished" podID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerID="2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6" exitCode=0
Jul 28 17:46:02.068296 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.068266    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerDied","Data":"2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6"}
Jul 28 17:46:02.068296 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.068275    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"
Jul 28 17:46:02.068805 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.068313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78" event={"ID":"2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3","Type":"ContainerDied","Data":"122621b3e841d633c79203e57d16448c540d445ed04eb661a1303f8d4e54c86b"}
Jul 28 17:46:02.068805 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.068339    2571 scope.go:117] "RemoveContainer" containerID="f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7"
Jul 28 17:46:02.075861 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.075845    2571 scope.go:117] "RemoveContainer" containerID="2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6"
Jul 28 17:46:02.082414 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.082394    2571 scope.go:117] "RemoveContainer" containerID="8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a"
Jul 28 17:46:02.087528 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.087507    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"]
Jul 28 17:46:02.089068 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.089055    2571 scope.go:117] "RemoveContainer" containerID="f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7"
Jul 28 17:46:02.089390 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:46:02.089368    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7\": container with ID starting with f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7 not found: ID does not exist" containerID="f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7"
Jul 28 17:46:02.089489 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.089397    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7"} err="failed to get container status \"f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7\": rpc error: code = NotFound desc = could not find container \"f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7\": container with ID starting with f750b9c09323f1708386e3b15a58cef95ffefe77bf692f795f3451294fc70aa7 not found: ID does not exist"
Jul 28 17:46:02.089489 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.089419    2571 scope.go:117] "RemoveContainer" containerID="2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6"
Jul 28 17:46:02.089690 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:46:02.089672    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6\": container with ID starting with 2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6 not found: ID does not exist" containerID="2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6"
Jul 28 17:46:02.089798 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.089699    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6"} err="failed to get container status \"2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6\": rpc error: code = NotFound desc = could not find container \"2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6\": container with ID starting with 2d61f07f12d20c4512b5c182f132e7d0a30212921f4322904b7820931fac30b6 not found: ID does not exist"
Jul 28 17:46:02.089798 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.089720    2571 scope.go:117] "RemoveContainer" containerID="8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a"
Jul 28 17:46:02.089970 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:46:02.089935    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a\": container with ID starting with 8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a not found: ID does not exist" containerID="8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a"
Jul 28 17:46:02.090041 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.090007    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a"} err="failed to get container status \"8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a\": rpc error: code = NotFound desc = could not find container \"8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a\": container with ID starting with 8662217866ef863256c5965189892866d4086bcae92cb71fadac389d10049b7a not found: ID does not exist"
Jul 28 17:46:02.091308 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:02.091289    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-7sq78"]
Jul 28 17:46:03.707110 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:46:03.707079    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" path="/var/lib/kubelet/pods/2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3/volumes"
Jul 28 17:47:44.023550 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023522    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"]
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023752    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="storage-initializer"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023763    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="storage-initializer"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023771    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023777    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023782    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kube-rbac-proxy"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023788    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kube-rbac-proxy"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023829    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kserve-container"
Jul 28 17:47:44.024006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.023836    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e36b9da-7ec4-4a2c-b8cd-3242dbc66eb3" containerName="kube-rbac-proxy"
Jul 28 17:47:44.026603 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.026588    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.029680 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.029655    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:47:44.029680 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.029673    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:47:44.029840 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.029656    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 17:47:44.029840 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.029821    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:47:44.030671 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.030653    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:47:44.030671 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.030665    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:47:44.030803 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.030707    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:47:44.040650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.040628    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"]
Jul 28 17:47:44.128543 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.128522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.128633 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.128547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.128673 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.128622    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvv7t\" (UniqueName: \"kubernetes.io/projected/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-api-access-wvv7t\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.128673 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.128655    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.128738 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.128698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.229815 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.229790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.229916 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.229818    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.229916 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.229846    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvv7t\" (UniqueName: \"kubernetes.io/projected/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-api-access-wvv7t\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.229916 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.229862    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.229916 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.229883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.230156 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:47:44.229962    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 17:47:44.230156 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:47:44.230015    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls podName:169d1266-7fdf-4b88-bb79-4812d8cf0c5f nodeName:}" failed. No retries permitted until 2026-07-28 17:47:44.729997133 +0000 UTC m=+2249.490022739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" (UID: "169d1266-7fdf-4b88-bb79-4812d8cf0c5f") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 28 17:47:44.230339 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.230308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.230516 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.230499    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.230555 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.230521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.240632 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.240616    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvv7t\" (UniqueName: \"kubernetes.io/projected/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-api-access-wvv7t\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.731651 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.731628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.733808 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.733791    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:44.935883 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:44.935852    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:45.068756 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:45.068734    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"]
Jul 28 17:47:45.071049 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:47:45.071025    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169d1266_7fdf_4b88_bb79_4812d8cf0c5f.slice/crio-1baeccfd0dbe0db439eb0f7e21fffbe47df0f13cb8b98c12de3183a88827ef00 WatchSource:0}: Error finding container 1baeccfd0dbe0db439eb0f7e21fffbe47df0f13cb8b98c12de3183a88827ef00: Status 404 returned error can't find the container with id 1baeccfd0dbe0db439eb0f7e21fffbe47df0f13cb8b98c12de3183a88827ef00
Jul 28 17:47:45.387214 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:45.387135    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerStarted","Data":"e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9"}
Jul 28 17:47:45.387374 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:45.387218    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerStarted","Data":"1baeccfd0dbe0db439eb0f7e21fffbe47df0f13cb8b98c12de3183a88827ef00"}
Jul 28 17:47:49.400555 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:49.400527    2571 generic.go:356] "Generic (PLEG): container finished" podID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerID="e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9" exitCode=0
Jul 28 17:47:49.400887 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:49.400591    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerDied","Data":"e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9"}
Jul 28 17:47:49.401677 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:49.401661    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:47:56.432067 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:56.432033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerStarted","Data":"6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50"}
Jul 28 17:47:56.432488 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:56.432091    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerStarted","Data":"1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598"}
Jul 28 17:47:56.432488 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:56.432274    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:56.468503 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:56.468449    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podStartSLOduration=2.785790633 podStartE2EDuration="12.468433136s" totalImagesPullingTime="6.059808288s" totalInitContainerRuntime="3.622834215s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:56.468278915 +0000 UTC m=+2261.228304539" watchObservedRunningTime="2026-07-28 17:47:56.468433136 +0000 UTC m=+2261.228458764"
Jul 28 17:47:57.435604 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:57.435579    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:47:57.436714 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:57.436681    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:47:58.438537 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:47:58.438503    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:48:03.442761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:03.442735    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:48:03.443291 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:03.443264    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:48:13.443660 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:13.443624    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:48:23.443467 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:23.443425    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:48:33.443276 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:33.443223    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:48:43.443297 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:43.443240    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:48:53.443843 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:48:53.443806    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:49:03.444060 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:03.444033    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:49:04.511410 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:04.511379    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"]
Jul 28 17:49:04.511764 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:04.511605    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container" containerID="cri-o://1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598" gracePeriod=30
Jul 28 17:49:04.511764 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:04.511678    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kube-rbac-proxy" containerID="cri-o://6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50" gracePeriod=30
Jul 28 17:49:04.659913 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:04.659887    2571 generic.go:356] "Generic (PLEG): container finished" podID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerID="6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50" exitCode=2
Jul 28 17:49:04.660004 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:04.659963    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerDied","Data":"6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50"}
Jul 28 17:49:07.947261 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:07.947223    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:49:08.083409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083342    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-rbac-proxy-sar-config\") pod \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") "
Jul 28 17:49:08.083409 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083406    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls\") pod \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") "
Jul 28 17:49:08.083553 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083434    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-cabundle-cert\") pod \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") "
Jul 28 17:49:08.083553 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083461    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvv7t\" (UniqueName: \"kubernetes.io/projected/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-api-access-wvv7t\") pod \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") "
Jul 28 17:49:08.083553 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083500    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kserve-provision-location\") pod \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\" (UID: \"169d1266-7fdf-4b88-bb79-4812d8cf0c5f\") "
Jul 28 17:49:08.083745 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083725    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "169d1266-7fdf-4b88-bb79-4812d8cf0c5f" (UID: "169d1266-7fdf-4b88-bb79-4812d8cf0c5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:49:08.083877 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083854    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "169d1266-7fdf-4b88-bb79-4812d8cf0c5f" (UID: "169d1266-7fdf-4b88-bb79-4812d8cf0c5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:49:08.083938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.083891    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "169d1266-7fdf-4b88-bb79-4812d8cf0c5f" (UID: "169d1266-7fdf-4b88-bb79-4812d8cf0c5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:08.085387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.085368    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "169d1266-7fdf-4b88-bb79-4812d8cf0c5f" (UID: "169d1266-7fdf-4b88-bb79-4812d8cf0c5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:08.085387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.085382    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-api-access-wvv7t" (OuterVolumeSpecName: "kube-api-access-wvv7t") pod "169d1266-7fdf-4b88-bb79-4812d8cf0c5f" (UID: "169d1266-7fdf-4b88-bb79-4812d8cf0c5f"). InnerVolumeSpecName "kube-api-access-wvv7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:08.183922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.183897    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.183922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.183922    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.184045 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.183932    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.184045 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.183942    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.184045 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.183950    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvv7t\" (UniqueName: \"kubernetes.io/projected/169d1266-7fdf-4b88-bb79-4812d8cf0c5f-kube-api-access-wvv7t\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.675320 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.675298    2571 generic.go:356] "Generic (PLEG): container finished" podID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerID="1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598" exitCode=0
Jul 28 17:49:08.675427 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.675329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerDied","Data":"1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598"}
Jul 28 17:49:08.675427 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.675371    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"
Jul 28 17:49:08.675427 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.675384    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4" event={"ID":"169d1266-7fdf-4b88-bb79-4812d8cf0c5f","Type":"ContainerDied","Data":"1baeccfd0dbe0db439eb0f7e21fffbe47df0f13cb8b98c12de3183a88827ef00"}
Jul 28 17:49:08.675427 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.675411    2571 scope.go:117] "RemoveContainer" containerID="6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50"
Jul 28 17:49:08.682989 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.682974    2571 scope.go:117] "RemoveContainer" containerID="1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598"
Jul 28 17:49:08.689299 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.689280    2571 scope.go:117] "RemoveContainer" containerID="e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9"
Jul 28 17:49:08.695775 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.695760    2571 scope.go:117] "RemoveContainer" containerID="6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50"
Jul 28 17:49:08.695988 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:49:08.695973    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50\": container with ID starting with 6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50 not found: ID does not exist" containerID="6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50"
Jul 28 17:49:08.696028 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.695995    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50"} err="failed to get container status \"6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50\": rpc error: code = NotFound desc = could not find container \"6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50\": container with ID starting with 6d994146043d81deed35492626e4753ad63c6e763b410260d15751b85a4f2d50 not found: ID does not exist"
Jul 28 17:49:08.696028 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.696010    2571 scope.go:117] "RemoveContainer" containerID="1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598"
Jul 28 17:49:08.696210 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:49:08.696190    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598\": container with ID starting with 1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598 not found: ID does not exist" containerID="1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598"
Jul 28 17:49:08.696280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.696216    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598"} err="failed to get container status \"1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598\": rpc error: code = NotFound desc = could not find container \"1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598\": container with ID starting with 1cf50c0809fe7e36161a5b6404e39c8fdb6b13b54602484315ec6d3dae46a598 not found: ID does not exist"
Jul 28 17:49:08.696280 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.696232    2571 scope.go:117] "RemoveContainer" containerID="e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9"
Jul 28 17:49:08.696452 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:49:08.696437    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9\": container with ID starting with e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9 not found: ID does not exist" containerID="e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9"
Jul 28 17:49:08.696493 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.696456    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9"} err="failed to get container status \"e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9\": rpc error: code = NotFound desc = could not find container \"e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9\": container with ID starting with e5f143968404a0ffce1c89ecd7a8fd40b65367f14abaedd7cd7ac2fed315c5c9 not found: ID does not exist"
Jul 28 17:49:08.721714 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.721690    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"]
Jul 28 17:49:08.735966 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:08.735943    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-f2zf4"]
Jul 28 17:49:09.706166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:49:09.706131    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" path="/var/lib/kubelet/pods/169d1266-7fdf-4b88-bb79-4812d8cf0c5f/volumes"
Jul 28 17:52:05.071005 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.070974    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"]
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071206    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071217    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071227    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="storage-initializer"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071232    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="storage-initializer"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071241    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kube-rbac-proxy"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071260    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kube-rbac-proxy"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071301    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kserve-container"
Jul 28 17:52:05.071412 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.071307    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="169d1266-7fdf-4b88-bb79-4812d8cf0c5f" containerName="kube-rbac-proxy"
Jul 28 17:52:05.074065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.074052    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.077080 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.077048    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:52:05.077513 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.077495    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:52:05.077636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.077558    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:52:05.078187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.078165    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 17:52:05.078295 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.078206    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:52:05.078295 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.078220    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 17:52:05.078295 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.078266    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:52:05.088299 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.088280    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"]
Jul 28 17:52:05.201922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.201899    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.202023 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.201938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzmq\" (UniqueName: \"kubernetes.io/projected/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-api-access-znzmq\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.202023 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.201956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.202101 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.202036    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.202101 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.202075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303148    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303180    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzmq\" (UniqueName: \"kubernetes.io/projected/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-api-access-znzmq\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303648 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303937 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.303986 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.303967    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.305584 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.305567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.312145 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.312125    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzmq\" (UniqueName: \"kubernetes.io/projected/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-api-access-znzmq\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.384166 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.384113    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:05.513066 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:05.513039    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"]
Jul 28 17:52:05.515304 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:52:05.515275    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3aeb74_8a48_4bae_95f3_f10381b3d31c.slice/crio-d19e0b141027975333e9a7af2c189bf9e74f90b702ea8869c9741c0025b9349d WatchSource:0}: Error finding container d19e0b141027975333e9a7af2c189bf9e74f90b702ea8869c9741c0025b9349d: Status 404 returned error can't find the container with id d19e0b141027975333e9a7af2c189bf9e74f90b702ea8869c9741c0025b9349d
Jul 28 17:52:06.221147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:06.221109    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerStarted","Data":"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"}
Jul 28 17:52:06.221567 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:06.221169    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerStarted","Data":"d19e0b141027975333e9a7af2c189bf9e74f90b702ea8869c9741c0025b9349d"}
Jul 28 17:52:10.233885 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:10.233854    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerID="c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f" exitCode=0
Jul 28 17:52:10.234210 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:10.233925    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerDied","Data":"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"}
Jul 28 17:52:11.239455 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:11.239420    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerStarted","Data":"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"}
Jul 28 17:52:11.239871 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:11.239526    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerStarted","Data":"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"}
Jul 28 17:52:11.239871 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:11.239814    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:11.263240 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:11.263196    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" podStartSLOduration=2.447517655 podStartE2EDuration="6.263185165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81566751s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:11.262383184 +0000 UTC m=+2516.022408827" watchObservedRunningTime="2026-07-28 17:52:11.263185165 +0000 UTC m=+2516.023210792"
Jul 28 17:52:12.242787 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:12.242751    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:12.244131 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:12.244105    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:52:13.245857 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:13.245813    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:52:18.249895 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:18.249865    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:18.250780 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:18.250764    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:25.216235 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:25.216204    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"]
Jul 28 17:52:25.216683 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:25.216467    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kserve-container" containerID="cri-o://8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e" gracePeriod=30
Jul 28 17:52:25.216683 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:25.216498    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kube-rbac-proxy" containerID="cri-o://ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f" gracePeriod=30
Jul 28 17:52:25.857580 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:25.857555    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:26.040126 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040064    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-rbac-proxy-sar-config\") pod \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") "
Jul 28 17:52:26.040126 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040100    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kserve-provision-location\") pod \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") "
Jul 28 17:52:26.040126 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040125    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znzmq\" (UniqueName: \"kubernetes.io/projected/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-api-access-znzmq\") pod \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") "
Jul 28 17:52:26.040395 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-proxy-tls\") pod \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") "
Jul 28 17:52:26.040395 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040179    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-cabundle-cert\") pod \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\" (UID: \"3c3aeb74-8a48-4bae-95f3-f10381b3d31c\") "
Jul 28 17:52:26.040497 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040403    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c3aeb74-8a48-4bae-95f3-f10381b3d31c" (UID: "3c3aeb74-8a48-4bae-95f3-f10381b3d31c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:26.040551 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c3aeb74-8a48-4bae-95f3-f10381b3d31c" (UID: "3c3aeb74-8a48-4bae-95f3-f10381b3d31c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:26.040614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.040595    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c3aeb74-8a48-4bae-95f3-f10381b3d31c" (UID: "3c3aeb74-8a48-4bae-95f3-f10381b3d31c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:26.042062 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.042036    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-api-access-znzmq" (OuterVolumeSpecName: "kube-api-access-znzmq") pod "3c3aeb74-8a48-4bae-95f3-f10381b3d31c" (UID: "3c3aeb74-8a48-4bae-95f3-f10381b3d31c"). InnerVolumeSpecName "kube-api-access-znzmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:26.042147 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.042106    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c3aeb74-8a48-4bae-95f3-f10381b3d31c" (UID: "3c3aeb74-8a48-4bae-95f3-f10381b3d31c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:26.140864 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.140840    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:52:26.140864 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.140859    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:52:26.140979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.140868    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzmq\" (UniqueName: \"kubernetes.io/projected/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-kube-api-access-znzmq\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:52:26.140979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.140878    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:52:26.140979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.140887    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c3aeb74-8a48-4bae-95f3-f10381b3d31c-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:52:26.292428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292377    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerID="ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f" exitCode=2
Jul 28 17:52:26.292428 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292396    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerID="8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e" exitCode=0
Jul 28 17:52:26.292743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292427    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"
Jul 28 17:52:26.292743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerDied","Data":"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"}
Jul 28 17:52:26.292743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292523    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerDied","Data":"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"}
Jul 28 17:52:26.292743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292547    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg" event={"ID":"3c3aeb74-8a48-4bae-95f3-f10381b3d31c","Type":"ContainerDied","Data":"d19e0b141027975333e9a7af2c189bf9e74f90b702ea8869c9741c0025b9349d"}
Jul 28 17:52:26.292743 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.292574    2571 scope.go:117] "RemoveContainer" containerID="ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"
Jul 28 17:52:26.300918 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.300903    2571 scope.go:117] "RemoveContainer" containerID="8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"
Jul 28 17:52:26.307778 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.307758    2571 scope.go:117] "RemoveContainer" containerID="c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"
Jul 28 17:52:26.314089 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314074    2571 scope.go:117] "RemoveContainer" containerID="ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"
Jul 28 17:52:26.314327 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:52:26.314311    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f\": container with ID starting with ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f not found: ID does not exist" containerID="ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"
Jul 28 17:52:26.314387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314335    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"} err="failed to get container status \"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f\": rpc error: code = NotFound desc = could not find container \"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f\": container with ID starting with ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f not found: ID does not exist"
Jul 28 17:52:26.314387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314352    2571 scope.go:117] "RemoveContainer" containerID="8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"
Jul 28 17:52:26.314572 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:52:26.314555    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e\": container with ID starting with 8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e not found: ID does not exist" containerID="8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"
Jul 28 17:52:26.314636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314581    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"} err="failed to get container status \"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e\": rpc error: code = NotFound desc = could not find container \"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e\": container with ID starting with 8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e not found: ID does not exist"
Jul 28 17:52:26.314636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314602    2571 scope.go:117] "RemoveContainer" containerID="c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"
Jul 28 17:52:26.314815 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:52:26.314798    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f\": container with ID starting with c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f not found: ID does not exist" containerID="c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"
Jul 28 17:52:26.314855 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314821    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"} err="failed to get container status \"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f\": rpc error: code = NotFound desc = could not find container \"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f\": container with ID starting with c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f not found: ID does not exist"
Jul 28 17:52:26.314855 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.314836    2571 scope.go:117] "RemoveContainer" containerID="ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"
Jul 28 17:52:26.315022 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.315006    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f"} err="failed to get container status \"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f\": rpc error: code = NotFound desc = could not find container \"ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f\": container with ID starting with ebae66572cb99e48b68c2308ab7dd348a525dc292ec70b847744db89fad0848f not found: ID does not exist"
Jul 28 17:52:26.315065 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.315023    2571 scope.go:117] "RemoveContainer" containerID="8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"
Jul 28 17:52:26.315205 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.315190    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e"} err="failed to get container status \"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e\": rpc error: code = NotFound desc = could not find container \"8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e\": container with ID starting with 8955ebe249978d449c64f990a35fda7f3daf94a2cc35b1a84dc844747387872e not found: ID does not exist"
Jul 28 17:52:26.315205 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.315204    2571 scope.go:117] "RemoveContainer" containerID="c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"
Jul 28 17:52:26.315532 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.315489    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f"} err="failed to get container status \"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f\": rpc error: code = NotFound desc = could not find container \"c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f\": container with ID starting with c6e8ee1a75d78e58a8b7320562f685145209aa73799ed447ab9e683f885e2a5f not found: ID does not exist"
Jul 28 17:52:26.317765 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.317746    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"]
Jul 28 17:52:26.321259 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:26.321224    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-p27xg"]
Jul 28 17:52:27.706467 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:52:27.706439    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" path="/var/lib/kubelet/pods/3c3aeb74-8a48-4bae-95f3-f10381b3d31c/volumes"
Jul 28 17:53:25.681579 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681545    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"]
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681906    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kube-rbac-proxy"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681922    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kube-rbac-proxy"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681933    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kserve-container"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681940    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kserve-container"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681961    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="storage-initializer"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.681969    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="storage-initializer"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.682028    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kube-rbac-proxy"
Jul 28 17:53:25.682064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.682041    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3aeb74-8a48-4bae-95f3-f10381b3d31c" containerName="kserve-container"
Jul 28 17:53:25.685145 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.685125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.688001 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.687974    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:53:25.688104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.688061    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:53:25.688104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.688078    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:53:25.688202 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.688188    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 17:53:25.688376 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.688361    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:53:25.688425 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.688361    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 17:53:25.689171 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.689155    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:53:25.709136 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.709115    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"]
Jul 28 17:53:25.832525 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.832500    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc58455-23ff-4e24-a87a-a75ab129da91-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.832611 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.832530    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc58455-23ff-4e24-a87a-a75ab129da91-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.832611 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.832564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.832686 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.832616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.832686 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.832650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krsx\" (UniqueName: \"kubernetes.io/projected/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-api-access-2krsx\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.933322 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.933274    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.933322 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.933300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.933322 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.933318    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krsx\" (UniqueName: \"kubernetes.io/projected/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-api-access-2krsx\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.933459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.933345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc58455-23ff-4e24-a87a-a75ab129da91-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.933459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.933441    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc58455-23ff-4e24-a87a-a75ab129da91-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.933769 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.933751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc58455-23ff-4e24-a87a-a75ab129da91-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.934020 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.934001    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.934063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.934001    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.935837 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.935818    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc58455-23ff-4e24-a87a-a75ab129da91-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.946976 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.946960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krsx\" (UniqueName: \"kubernetes.io/projected/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-api-access-2krsx\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:25.994917 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:25.994896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:26.131485 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:26.131456    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"]
Jul 28 17:53:26.133723 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:53:26.133700    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc58455_23ff_4e24_a87a_a75ab129da91.slice/crio-787f4a4edc5382de05dcce3adef95d4fa9c4ea7dc9df9675ec3954305c0caf6f WatchSource:0}: Error finding container 787f4a4edc5382de05dcce3adef95d4fa9c4ea7dc9df9675ec3954305c0caf6f: Status 404 returned error can't find the container with id 787f4a4edc5382de05dcce3adef95d4fa9c4ea7dc9df9675ec3954305c0caf6f
Jul 28 17:53:26.478848 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:26.478816    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerStarted","Data":"d13e6559736ca4a48a471f487fd3996a20f9f6b09b6ffbc81b8805bf892009bd"}
Jul 28 17:53:26.479004 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:26.478883    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerStarted","Data":"787f4a4edc5382de05dcce3adef95d4fa9c4ea7dc9df9675ec3954305c0caf6f"}
Jul 28 17:53:30.494229 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:30.494201    2571 generic.go:356] "Generic (PLEG): container finished" podID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerID="d13e6559736ca4a48a471f487fd3996a20f9f6b09b6ffbc81b8805bf892009bd" exitCode=0
Jul 28 17:53:30.494556 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:30.494273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerDied","Data":"d13e6559736ca4a48a471f487fd3996a20f9f6b09b6ffbc81b8805bf892009bd"}
Jul 28 17:53:30.553861 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:30.553843    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:53:31.502074 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:31.502026    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerStarted","Data":"9498fe51946746c5eb770066158d19ad5c4fbcedc0b24151bba7a119cb651d0b"}
Jul 28 17:53:33.511201 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:33.511170    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerStarted","Data":"31f17d15242be36004feb41411b48f077b7644b29786c764450a6241c57e7cf3"}
Jul 28 17:53:33.511589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:33.511236    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerStarted","Data":"0596cb1a9144234755d169a3b6291eaffded0f3e544f8f116f919caf9725e977"}
Jul 28 17:53:33.511589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:33.511375    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:33.511589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:33.511433    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:33.511589 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:33.511458    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:53:33.539051 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:33.539011    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podStartSLOduration=2.823052032 podStartE2EDuration="8.538996975s" totalImagesPullingTime="1.997714827s" totalInitContainerRuntime="3.718230116s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:33.537357949 +0000 UTC m=+2598.297383576" watchObservedRunningTime="2026-07-28 17:53:33.538996975 +0000 UTC m=+2598.299022602"
Jul 28 17:53:39.520148 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:53:39.520081    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:54:09.521905 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:09.521866    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:54:39.522837 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:39.522810    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:54:45.746500 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:45.746468    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"]
Jul 28 17:54:45.747006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:45.746758    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container" containerID="cri-o://9498fe51946746c5eb770066158d19ad5c4fbcedc0b24151bba7a119cb651d0b" gracePeriod=30
Jul 28 17:54:45.747006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:45.746792    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-agent" containerID="cri-o://0596cb1a9144234755d169a3b6291eaffded0f3e544f8f116f919caf9725e977" gracePeriod=30
Jul 28 17:54:45.747006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:45.746911    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" containerID="cri-o://31f17d15242be36004feb41411b48f077b7644b29786c764450a6241c57e7cf3" gracePeriod=30
Jul 28 17:54:46.755761 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:46.755727    2571 generic.go:356] "Generic (PLEG): container finished" podID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerID="31f17d15242be36004feb41411b48f077b7644b29786c764450a6241c57e7cf3" exitCode=2
Jul 28 17:54:46.756207 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:46.755804    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerDied","Data":"31f17d15242be36004feb41411b48f077b7644b29786c764450a6241c57e7cf3"}
Jul 28 17:54:47.760958 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:47.760932    2571 generic.go:356] "Generic (PLEG): container finished" podID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerID="9498fe51946746c5eb770066158d19ad5c4fbcedc0b24151bba7a119cb651d0b" exitCode=0
Jul 28 17:54:47.761319 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:47.760997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerDied","Data":"9498fe51946746c5eb770066158d19ad5c4fbcedc0b24151bba7a119cb651d0b"}
Jul 28 17:54:49.515359 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:49.515322    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:54:49.520636 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:49.520613    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:54:54.515685 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:54.515644    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:54:59.515954 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:59.515913    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:54:59.516349 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:59.516043    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:54:59.521391 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:54:59.521364    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:55:04.516138 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:04.516105    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:55:09.515896 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:09.515813    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:55:09.521434 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:09.521410    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:55:09.521548 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:09.521536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:55:14.515575 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:14.515538    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:55:15.847646 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.847615    2571 generic.go:356] "Generic (PLEG): container finished" podID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerID="0596cb1a9144234755d169a3b6291eaffded0f3e544f8f116f919caf9725e977" exitCode=137
Jul 28 17:55:15.847938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.847681    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerDied","Data":"0596cb1a9144234755d169a3b6291eaffded0f3e544f8f116f919caf9725e977"}
Jul 28 17:55:15.911979 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.911958    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:55:15.946146 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946119    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc58455-23ff-4e24-a87a-a75ab129da91-kserve-provision-location\") pod \"ffc58455-23ff-4e24-a87a-a75ab129da91\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") "
Jul 28 17:55:15.946233 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946194    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-rbac-proxy-sar-config\") pod \"ffc58455-23ff-4e24-a87a-a75ab129da91\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") "
Jul 28 17:55:15.946300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946269    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-cabundle-cert\") pod \"ffc58455-23ff-4e24-a87a-a75ab129da91\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") "
Jul 28 17:55:15.946345 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946301    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2krsx\" (UniqueName: \"kubernetes.io/projected/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-api-access-2krsx\") pod \"ffc58455-23ff-4e24-a87a-a75ab129da91\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") "
Jul 28 17:55:15.946390 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946350    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc58455-23ff-4e24-a87a-a75ab129da91-proxy-tls\") pod \"ffc58455-23ff-4e24-a87a-a75ab129da91\" (UID: \"ffc58455-23ff-4e24-a87a-a75ab129da91\") "
Jul 28 17:55:15.946491 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946439    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc58455-23ff-4e24-a87a-a75ab129da91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffc58455-23ff-4e24-a87a-a75ab129da91" (UID: "ffc58455-23ff-4e24-a87a-a75ab129da91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:15.946619 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946575    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ffc58455-23ff-4e24-a87a-a75ab129da91" (UID: "ffc58455-23ff-4e24-a87a-a75ab129da91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:15.946688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946627    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffc58455-23ff-4e24-a87a-a75ab129da91-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:55:15.946688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.946650    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffc58455-23ff-4e24-a87a-a75ab129da91" (UID: "ffc58455-23ff-4e24-a87a-a75ab129da91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:55:15.948416 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.948391    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc58455-23ff-4e24-a87a-a75ab129da91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffc58455-23ff-4e24-a87a-a75ab129da91" (UID: "ffc58455-23ff-4e24-a87a-a75ab129da91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:15.948522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:15.948448    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-api-access-2krsx" (OuterVolumeSpecName: "kube-api-access-2krsx") pod "ffc58455-23ff-4e24-a87a-a75ab129da91" (UID: "ffc58455-23ff-4e24-a87a-a75ab129da91"). InnerVolumeSpecName "kube-api-access-2krsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:16.047116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.047068    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:55:16.047116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.047086    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2krsx\" (UniqueName: \"kubernetes.io/projected/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-api-access-2krsx\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:55:16.047116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.047095    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffc58455-23ff-4e24-a87a-a75ab129da91-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:55:16.047116 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.047105    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffc58455-23ff-4e24-a87a-a75ab129da91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:55:16.852470 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.852447    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x" event={"ID":"ffc58455-23ff-4e24-a87a-a75ab129da91","Type":"ContainerDied","Data":"787f4a4edc5382de05dcce3adef95d4fa9c4ea7dc9df9675ec3954305c0caf6f"}
Jul 28 17:55:16.852899 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.852488    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"
Jul 28 17:55:16.852899 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.852511    2571 scope.go:117] "RemoveContainer" containerID="31f17d15242be36004feb41411b48f077b7644b29786c764450a6241c57e7cf3"
Jul 28 17:55:16.860426 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.860183    2571 scope.go:117] "RemoveContainer" containerID="0596cb1a9144234755d169a3b6291eaffded0f3e544f8f116f919caf9725e977"
Jul 28 17:55:16.866689 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.866672    2571 scope.go:117] "RemoveContainer" containerID="9498fe51946746c5eb770066158d19ad5c4fbcedc0b24151bba7a119cb651d0b"
Jul 28 17:55:16.872922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.872904    2571 scope.go:117] "RemoveContainer" containerID="d13e6559736ca4a48a471f487fd3996a20f9f6b09b6ffbc81b8805bf892009bd"
Jul 28 17:55:16.879697 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.879680    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"]
Jul 28 17:55:16.884962 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:16.884943    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-88g7x"]
Jul 28 17:55:17.711115 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:55:17.708052    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" path="/var/lib/kubelet/pods/ffc58455-23ff-4e24-a87a-a75ab129da91/volumes"
Jul 28 17:56:07.833832 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.833797    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"]
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834022    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="storage-initializer"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834032    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="storage-initializer"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834046    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834052    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834058    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-agent"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834063    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-agent"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834069    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834074    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834113    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-agent"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834120    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kserve-container"
Jul 28 17:56:07.834216 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.834127    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc58455-23ff-4e24-a87a-a75ab129da91" containerName="kube-rbac-proxy"
Jul 28 17:56:07.836808 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.836794    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:07.839347 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839325    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:56:07.839475 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839362    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:56:07.839475 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839371    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:56:07.839722 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839706    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 17:56:07.839811 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839718    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:56:07.839811 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839744    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:56:07.839915 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.839810    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:56:07.848271 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.848234    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"]
Jul 28 17:56:07.976413 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.976387    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddp2z\" (UniqueName: \"kubernetes.io/projected/ebae9792-77bb-46ec-a92f-10bd75487351-kube-api-access-ddp2z\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:07.976536 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.976421    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:07.976536 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.976446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:07.976536 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.976504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebae9792-77bb-46ec-a92f-10bd75487351-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:07.976536 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:07.976529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebae9792-77bb-46ec-a92f-10bd75487351-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.077683 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.077660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.077796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.077691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.077796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.077714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebae9792-77bb-46ec-a92f-10bd75487351-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.077796 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.077732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebae9792-77bb-46ec-a92f-10bd75487351-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.077949 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.077846    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddp2z\" (UniqueName: \"kubernetes.io/projected/ebae9792-77bb-46ec-a92f-10bd75487351-kube-api-access-ddp2z\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.078086 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.078070    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebae9792-77bb-46ec-a92f-10bd75487351-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.078334 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.078304    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.078418 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.078403    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.079985 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.079968    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebae9792-77bb-46ec-a92f-10bd75487351-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.087614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.087561    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddp2z\" (UniqueName: \"kubernetes.io/projected/ebae9792-77bb-46ec-a92f-10bd75487351-kube-api-access-ddp2z\") pod \"isvc-paddle-runtime-predictor-58b99fff76-bdmx9\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.146920 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.146894    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:08.272638 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:08.272611    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"]
Jul 28 17:56:08.274994 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:56:08.274967    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebae9792_77bb_46ec_a92f_10bd75487351.slice/crio-a7d122371e940857cfb795df0d9406555e69ead681fbf570718f7e990a38a36e WatchSource:0}: Error finding container a7d122371e940857cfb795df0d9406555e69ead681fbf570718f7e990a38a36e: Status 404 returned error can't find the container with id a7d122371e940857cfb795df0d9406555e69ead681fbf570718f7e990a38a36e
Jul 28 17:56:09.018554 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:09.018519    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerStarted","Data":"9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9"}
Jul 28 17:56:09.018986 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:09.018573    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerStarted","Data":"a7d122371e940857cfb795df0d9406555e69ead681fbf570718f7e990a38a36e"}
Jul 28 17:56:13.032862 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:13.032834    2571 generic.go:356] "Generic (PLEG): container finished" podID="ebae9792-77bb-46ec-a92f-10bd75487351" containerID="9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9" exitCode=0
Jul 28 17:56:13.033190 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:13.032911    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerDied","Data":"9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9"}
Jul 28 17:56:24.077226 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:24.077194    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerStarted","Data":"c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8"}
Jul 28 17:56:24.077627 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:24.077275    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerStarted","Data":"9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f"}
Jul 28 17:56:24.077627 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:24.077457    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:24.100874 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:24.100830    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podStartSLOduration=2.710202652 podStartE2EDuration="17.10081646s" totalImagesPullingTime="10.140515615s" totalInitContainerRuntime="4.250098193s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:24.096769662 +0000 UTC m=+2768.856795288" watchObservedRunningTime="2026-07-28 17:56:24.10081646 +0000 UTC m=+2768.860842089"
Jul 28 17:56:25.080400 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:25.080363    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:25.081438 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:25.081414    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:56:26.084504 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:26.084456    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:56:31.088960 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:31.088933    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:56:31.089372 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:31.089348    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:56:41.089554 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:41.089474    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:56:51.090396 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:56:51.090358    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:57:01.089547 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:01.089514    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:57:11.090054 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:11.090026    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:57:19.206532 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:19.206504    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"]
Jul 28 17:57:19.206964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:19.206779    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" containerID="cri-o://9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f" gracePeriod=30
Jul 28 17:57:19.206964 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:19.206791    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kube-rbac-proxy" containerID="cri-o://c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8" gracePeriod=30
Jul 28 17:57:20.260387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:20.260354    2571 generic.go:356] "Generic (PLEG): container finished" podID="ebae9792-77bb-46ec-a92f-10bd75487351" containerID="c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8" exitCode=2
Jul 28 17:57:20.260744 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:20.260397    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerDied","Data":"c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8"}
Jul 28 17:57:21.084600 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.084554    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 28 17:57:21.090025 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.090002    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:57:21.463620 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.463598    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:57:21.486974 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.486950    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-kube-rbac-proxy-sar-config\") pod \"ebae9792-77bb-46ec-a92f-10bd75487351\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") "
Jul 28 17:57:21.487083 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.487002    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebae9792-77bb-46ec-a92f-10bd75487351-kserve-provision-location\") pod \"ebae9792-77bb-46ec-a92f-10bd75487351\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") "
Jul 28 17:57:21.487083 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.487025    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebae9792-77bb-46ec-a92f-10bd75487351-proxy-tls\") pod \"ebae9792-77bb-46ec-a92f-10bd75487351\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") "
Jul 28 17:57:21.487083 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.487063    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-cabundle-cert\") pod \"ebae9792-77bb-46ec-a92f-10bd75487351\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") "
Jul 28 17:57:21.487233 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.487098    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddp2z\" (UniqueName: \"kubernetes.io/projected/ebae9792-77bb-46ec-a92f-10bd75487351-kube-api-access-ddp2z\") pod \"ebae9792-77bb-46ec-a92f-10bd75487351\" (UID: \"ebae9792-77bb-46ec-a92f-10bd75487351\") "
Jul 28 17:57:21.487423 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.487395    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebae9792-77bb-46ec-a92f-10bd75487351" (UID: "ebae9792-77bb-46ec-a92f-10bd75487351"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:57:21.487538 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.487501    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebae9792-77bb-46ec-a92f-10bd75487351" (UID: "ebae9792-77bb-46ec-a92f-10bd75487351"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:57:21.489459 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.489427    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebae9792-77bb-46ec-a92f-10bd75487351-kube-api-access-ddp2z" (OuterVolumeSpecName: "kube-api-access-ddp2z") pod "ebae9792-77bb-46ec-a92f-10bd75487351" (UID: "ebae9792-77bb-46ec-a92f-10bd75487351"). InnerVolumeSpecName "kube-api-access-ddp2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:57:21.489601 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.489582    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebae9792-77bb-46ec-a92f-10bd75487351-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebae9792-77bb-46ec-a92f-10bd75487351" (UID: "ebae9792-77bb-46ec-a92f-10bd75487351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:57:21.500570 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.500542    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebae9792-77bb-46ec-a92f-10bd75487351-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebae9792-77bb-46ec-a92f-10bd75487351" (UID: "ebae9792-77bb-46ec-a92f-10bd75487351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:21.588027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.588005    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:57:21.588027 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.588026    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddp2z\" (UniqueName: \"kubernetes.io/projected/ebae9792-77bb-46ec-a92f-10bd75487351-kube-api-access-ddp2z\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:57:21.588152 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.588036    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebae9792-77bb-46ec-a92f-10bd75487351-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:57:21.588152 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.588044    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebae9792-77bb-46ec-a92f-10bd75487351-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:57:21.588152 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:21.588053    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebae9792-77bb-46ec-a92f-10bd75487351-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:57:22.267629 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.267606    2571 generic.go:356] "Generic (PLEG): container finished" podID="ebae9792-77bb-46ec-a92f-10bd75487351" containerID="9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f" exitCode=0
Jul 28 17:57:22.267731 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.267651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerDied","Data":"9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f"}
Jul 28 17:57:22.267731 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.267659    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"
Jul 28 17:57:22.267731 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.267695    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9" event={"ID":"ebae9792-77bb-46ec-a92f-10bd75487351","Type":"ContainerDied","Data":"a7d122371e940857cfb795df0d9406555e69ead681fbf570718f7e990a38a36e"}
Jul 28 17:57:22.267858 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.267746    2571 scope.go:117] "RemoveContainer" containerID="c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8"
Jul 28 17:57:22.275104 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.275088    2571 scope.go:117] "RemoveContainer" containerID="9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f"
Jul 28 17:57:22.281875 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.281859    2571 scope.go:117] "RemoveContainer" containerID="9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9"
Jul 28 17:57:22.288063 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.288040    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"]
Jul 28 17:57:22.288387 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.288369    2571 scope.go:117] "RemoveContainer" containerID="c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8"
Jul 28 17:57:22.288626 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:57:22.288610    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8\": container with ID starting with c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8 not found: ID does not exist" containerID="c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8"
Jul 28 17:57:22.288689 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.288634    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8"} err="failed to get container status \"c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8\": rpc error: code = NotFound desc = could not find container \"c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8\": container with ID starting with c3d83c0f8da211b8a8a10ab3553dd55ba2e48195082cd3dfcf0d06dd6200a5b8 not found: ID does not exist"
Jul 28 17:57:22.288689 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.288649    2571 scope.go:117] "RemoveContainer" containerID="9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f"
Jul 28 17:57:22.288867 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:57:22.288851    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f\": container with ID starting with 9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f not found: ID does not exist" containerID="9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f"
Jul 28 17:57:22.288910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.288872    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f"} err="failed to get container status \"9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f\": rpc error: code = NotFound desc = could not find container \"9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f\": container with ID starting with 9c492377783955c116ec47b6cbbb689e5b147940a9a76d59e13db928f829283f not found: ID does not exist"
Jul 28 17:57:22.288910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.288887    2571 scope.go:117] "RemoveContainer" containerID="9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9"
Jul 28 17:57:22.289094 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:57:22.289077    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9\": container with ID starting with 9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9 not found: ID does not exist" containerID="9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9"
Jul 28 17:57:22.289141 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.289096    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9"} err="failed to get container status \"9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9\": rpc error: code = NotFound desc = could not find container \"9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9\": container with ID starting with 9b2a453049b0cb47722e5620db96039fe7bc4c8254d1e9e620a7b3ca2eec42d9 not found: ID does not exist"
Jul 28 17:57:22.291900 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:22.291881    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-bdmx9"]
Jul 28 17:57:23.706603 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:23.706573    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" path="/var/lib/kubelet/pods/ebae9792-77bb-46ec-a92f-10bd75487351/volumes"
Jul 28 17:57:24.400158 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400126    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"]
Jul 28 17:57:24.400435 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400420    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container"
Jul 28 17:57:24.400492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400437    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container"
Jul 28 17:57:24.400492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400449    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kube-rbac-proxy"
Jul 28 17:57:24.400492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400454    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kube-rbac-proxy"
Jul 28 17:57:24.400492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400464    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="storage-initializer"
Jul 28 17:57:24.400492 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400469    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="storage-initializer"
Jul 28 17:57:24.400640 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400513    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kube-rbac-proxy"
Jul 28 17:57:24.400640 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.400522    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebae9792-77bb-46ec-a92f-10bd75487351" containerName="kserve-container"
Jul 28 17:57:24.404187 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.404172    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.406764 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.406740    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:57:24.406901 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.406764    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 17:57:24.406901 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.406778    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:57:24.406901 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.406787    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:57:24.406901 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.406896    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 17:57:24.407060 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.406915    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:57:24.407170 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.407158    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:57:24.413772 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.413753    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"]
Jul 28 17:57:24.509405 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.509383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.509546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.509409    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.509546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.509427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhnn\" (UniqueName: \"kubernetes.io/projected/894d8de5-b053-4b22-b6b7-0962f434924d-kube-api-access-wjhnn\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.509546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.509457    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/894d8de5-b053-4b22-b6b7-0962f434924d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.509648 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.509538    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/894d8de5-b053-4b22-b6b7-0962f434924d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.610731 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.610707    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/894d8de5-b053-4b22-b6b7-0962f434924d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.610818 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.610740    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.610818 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.610759    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.610818 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.610777    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhnn\" (UniqueName: \"kubernetes.io/projected/894d8de5-b053-4b22-b6b7-0962f434924d-kube-api-access-wjhnn\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.610818 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.610813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/894d8de5-b053-4b22-b6b7-0962f434924d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.611240 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.611221    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/894d8de5-b053-4b22-b6b7-0962f434924d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.611545 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.611525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.611593 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.611528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.613064 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.613044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/894d8de5-b053-4b22-b6b7-0962f434924d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.618889 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.618868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhnn\" (UniqueName: \"kubernetes.io/projected/894d8de5-b053-4b22-b6b7-0962f434924d-kube-api-access-wjhnn\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.714318 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.714267    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:24.833958 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:24.833935    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"]
Jul 28 17:57:24.836352 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:57:24.836324    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894d8de5_b053_4b22_b6b7_0962f434924d.slice/crio-8653836c3aee5333803c81fb400635d11f42d21c23d0d9fa3a78275512da2656 WatchSource:0}: Error finding container 8653836c3aee5333803c81fb400635d11f42d21c23d0d9fa3a78275512da2656: Status 404 returned error can't find the container with id 8653836c3aee5333803c81fb400635d11f42d21c23d0d9fa3a78275512da2656
Jul 28 17:57:25.280419 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:25.280387    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerStarted","Data":"8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a"}
Jul 28 17:57:25.280571 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:25.280442    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerStarted","Data":"8653836c3aee5333803c81fb400635d11f42d21c23d0d9fa3a78275512da2656"}
Jul 28 17:57:29.294816 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:29.294749    2571 generic.go:356] "Generic (PLEG): container finished" podID="894d8de5-b053-4b22-b6b7-0962f434924d" containerID="8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a" exitCode=0
Jul 28 17:57:29.294816 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:29.294785    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerDied","Data":"8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a"}
Jul 28 17:57:30.300848 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:30.300815    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerStarted","Data":"84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b"}
Jul 28 17:57:30.301225 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:30.300880    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerStarted","Data":"7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160"}
Jul 28 17:57:30.301225 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:30.301087    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:30.322994 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:30.322947    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podStartSLOduration=2.2213568439999998 podStartE2EDuration="6.322933409s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.101576565s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:30.321952675 +0000 UTC m=+2835.081978303" watchObservedRunningTime="2026-07-28 17:57:30.322933409 +0000 UTC m=+2835.082959033"
Jul 28 17:57:31.304595 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:31.304563    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:31.305742 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:31.305715    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:57:32.308136 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:32.308093    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:57:37.311691 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:37.311666    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:57:37.312132 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:37.312101    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:57:47.312417 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:47.312378    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:57:57.312718 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:57:57.312680    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:58:07.312117 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:07.312081    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:58:17.312458 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:17.312426    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:58:26.122948 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:26.122917    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"]
Jul 28 17:58:26.123433 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:26.123146    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" containerID="cri-o://7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160" gracePeriod=30
Jul 28 17:58:26.123433 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:26.123198    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kube-rbac-proxy" containerID="cri-o://84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b" gracePeriod=30
Jul 28 17:58:26.481916 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:26.481890    2571 generic.go:356] "Generic (PLEG): container finished" podID="894d8de5-b053-4b22-b6b7-0962f434924d" containerID="84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b" exitCode=2
Jul 28 17:58:26.482069 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:26.481922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerDied","Data":"84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b"}
Jul 28 17:58:27.308680 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:27.308641    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 17:58:27.312984 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:27.312964    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:58:28.362128 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.362108    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:58:28.490033 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.489973    2571 generic.go:356] "Generic (PLEG): container finished" podID="894d8de5-b053-4b22-b6b7-0962f434924d" containerID="7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160" exitCode=0
Jul 28 17:58:28.490033 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.490022    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"
Jul 28 17:58:28.490215 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.490053    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerDied","Data":"7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160"}
Jul 28 17:58:28.490215 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.490119    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7" event={"ID":"894d8de5-b053-4b22-b6b7-0962f434924d","Type":"ContainerDied","Data":"8653836c3aee5333803c81fb400635d11f42d21c23d0d9fa3a78275512da2656"}
Jul 28 17:58:28.490215 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.490155    2571 scope.go:117] "RemoveContainer" containerID="84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b"
Jul 28 17:58:28.499289 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.499273    2571 scope.go:117] "RemoveContainer" containerID="7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160"
Jul 28 17:58:28.506284 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.506268    2571 scope.go:117] "RemoveContainer" containerID="8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a"
Jul 28 17:58:28.512617 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.512603    2571 scope.go:117] "RemoveContainer" containerID="84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b"
Jul 28 17:58:28.512875 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:58:28.512858    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b\": container with ID starting with 84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b not found: ID does not exist" containerID="84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b"
Jul 28 17:58:28.512947 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.512882    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b"} err="failed to get container status \"84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b\": rpc error: code = NotFound desc = could not find container \"84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b\": container with ID starting with 84725a87724d44fd5025c4717059286a88cd33bb2e0f2ace9d437a2d66950b3b not found: ID does not exist"
Jul 28 17:58:28.512947 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.512900    2571 scope.go:117] "RemoveContainer" containerID="7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160"
Jul 28 17:58:28.513123 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:58:28.513107    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160\": container with ID starting with 7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160 not found: ID does not exist" containerID="7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160"
Jul 28 17:58:28.513165 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.513130    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160"} err="failed to get container status \"7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160\": rpc error: code = NotFound desc = could not find container \"7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160\": container with ID starting with 7bc722acc2438ffd9e50f8390cf2b76ebd27bf316bf56b16cf288fc92d874160 not found: ID does not exist"
Jul 28 17:58:28.513165 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.513144    2571 scope.go:117] "RemoveContainer" containerID="8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a"
Jul 28 17:58:28.513384 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:58:28.513367    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a\": container with ID starting with 8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a not found: ID does not exist" containerID="8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a"
Jul 28 17:58:28.513433 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.513389    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a"} err="failed to get container status \"8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a\": rpc error: code = NotFound desc = could not find container \"8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a\": container with ID starting with 8f3309e5929e33be000b66a73c4230b73f5724a5a836be9a8d0704bb6e47bd8a not found: ID does not exist"
Jul 28 17:58:28.532684 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.532669    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-kube-rbac-proxy-sar-config\") pod \"894d8de5-b053-4b22-b6b7-0962f434924d\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") "
Jul 28 17:58:28.532736 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.532697    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/894d8de5-b053-4b22-b6b7-0962f434924d-kserve-provision-location\") pod \"894d8de5-b053-4b22-b6b7-0962f434924d\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") "
Jul 28 17:58:28.532736 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.532718    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-cabundle-cert\") pod \"894d8de5-b053-4b22-b6b7-0962f434924d\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") "
Jul 28 17:58:28.532802 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.532749    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjhnn\" (UniqueName: \"kubernetes.io/projected/894d8de5-b053-4b22-b6b7-0962f434924d-kube-api-access-wjhnn\") pod \"894d8de5-b053-4b22-b6b7-0962f434924d\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") "
Jul 28 17:58:28.532802 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.532774    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/894d8de5-b053-4b22-b6b7-0962f434924d-proxy-tls\") pod \"894d8de5-b053-4b22-b6b7-0962f434924d\" (UID: \"894d8de5-b053-4b22-b6b7-0962f434924d\") "
Jul 28 17:58:28.533038 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.533017    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "894d8de5-b053-4b22-b6b7-0962f434924d" (UID: "894d8de5-b053-4b22-b6b7-0962f434924d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:28.533038 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.533029    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "894d8de5-b053-4b22-b6b7-0962f434924d" (UID: "894d8de5-b053-4b22-b6b7-0962f434924d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:28.534645 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.534614    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894d8de5-b053-4b22-b6b7-0962f434924d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "894d8de5-b053-4b22-b6b7-0962f434924d" (UID: "894d8de5-b053-4b22-b6b7-0962f434924d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:28.534714 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.534614    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894d8de5-b053-4b22-b6b7-0962f434924d-kube-api-access-wjhnn" (OuterVolumeSpecName: "kube-api-access-wjhnn") pod "894d8de5-b053-4b22-b6b7-0962f434924d" (UID: "894d8de5-b053-4b22-b6b7-0962f434924d"). InnerVolumeSpecName "kube-api-access-wjhnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:28.541825 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.541805    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d8de5-b053-4b22-b6b7-0962f434924d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "894d8de5-b053-4b22-b6b7-0962f434924d" (UID: "894d8de5-b053-4b22-b6b7-0962f434924d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:28.634195 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.634165    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:58:28.634195 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.634194    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjhnn\" (UniqueName: \"kubernetes.io/projected/894d8de5-b053-4b22-b6b7-0962f434924d-kube-api-access-wjhnn\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:58:28.634334 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.634204    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/894d8de5-b053-4b22-b6b7-0962f434924d-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:58:28.634334 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.634214    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/894d8de5-b053-4b22-b6b7-0962f434924d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:58:28.634334 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.634223    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/894d8de5-b053-4b22-b6b7-0962f434924d-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:58:28.812931 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.812907    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"]
Jul 28 17:58:28.817658 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:28.817641    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5cpv7"]
Jul 28 17:58:29.707011 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:29.706984    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" path="/var/lib/kubelet/pods/894d8de5-b053-4b22-b6b7-0962f434924d/volumes"
Jul 28 17:58:31.323667 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.323636    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"]
Jul 28 17:58:31.324474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324454    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kube-rbac-proxy"
Jul 28 17:58:31.324474 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324476    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kube-rbac-proxy"
Jul 28 17:58:31.324647 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324509    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container"
Jul 28 17:58:31.324647 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324518    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container"
Jul 28 17:58:31.324647 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324546    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="storage-initializer"
Jul 28 17:58:31.324647 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324554    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="storage-initializer"
Jul 28 17:58:31.324839 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324718    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kserve-container"
Jul 28 17:58:31.324839 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.324738    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="894d8de5-b053-4b22-b6b7-0962f434924d" containerName="kube-rbac-proxy"
Jul 28 17:58:31.328880 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.328860    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.332030 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.332004    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:58:31.332140 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.332041    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 17:58:31.332140 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.332082    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 17:58:31.332140 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.332098    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:58:31.332307 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.332210    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:58:31.333129 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.333114    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:58:31.333359 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.333345    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:58:31.337227 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.337207    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"]
Jul 28 17:58:31.451559 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.451522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b018744-0116-4331-bd03-c0b897f3f7f3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.451664 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.451568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk57k\" (UniqueName: \"kubernetes.io/projected/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-api-access-xk57k\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.451664 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.451593    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.451664 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.451649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b018744-0116-4331-bd03-c0b897f3f7f3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.451783 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.451695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552186 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b018744-0116-4331-bd03-c0b897f3f7f3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552209    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b018744-0116-4331-bd03-c0b897f3f7f3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552294 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk57k\" (UniqueName: \"kubernetes.io/projected/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-api-access-xk57k\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552419 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552296    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552546 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b018744-0116-4331-bd03-c0b897f3f7f3-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.552938 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.553006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.552948    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.554650 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.554632    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b018744-0116-4331-bd03-c0b897f3f7f3-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.561522 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.561501    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk57k\" (UniqueName: \"kubernetes.io/projected/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-api-access-xk57k\") pod \"isvc-pmml-predictor-8674795f59-65rhb\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.639022 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.639002    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:31.762824 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:31.762796    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"]
Jul 28 17:58:31.765771 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:58:31.765749    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b018744_0116_4331_bd03_c0b897f3f7f3.slice/crio-b4d9b9c7b4bb8461fafebd4df104d9d7a70951c30221006181ead2545de0bfef WatchSource:0}: Error finding container b4d9b9c7b4bb8461fafebd4df104d9d7a70951c30221006181ead2545de0bfef: Status 404 returned error can't find the container with id b4d9b9c7b4bb8461fafebd4df104d9d7a70951c30221006181ead2545de0bfef
Jul 28 17:58:32.505180 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:32.505147    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerStarted","Data":"a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794"}
Jul 28 17:58:32.505615 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:32.505205    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerStarted","Data":"b4d9b9c7b4bb8461fafebd4df104d9d7a70951c30221006181ead2545de0bfef"}
Jul 28 17:58:36.520154 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:36.520125    2571 generic.go:356] "Generic (PLEG): container finished" podID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerID="a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794" exitCode=0
Jul 28 17:58:36.520543 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:36.520165    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerDied","Data":"a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794"}
Jul 28 17:58:36.521173 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:36.521157    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:58:43.550293 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:43.550257    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerStarted","Data":"dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe"}
Jul 28 17:58:43.550679 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:43.550322    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerStarted","Data":"771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada"}
Jul 28 17:58:43.550679 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:43.550505    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:43.571732 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:43.571689    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podStartSLOduration=2.379037362 podStartE2EDuration="12.571676768s" totalImagesPullingTime="6.430900537s" totalInitContainerRuntime="3.761738869s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:43.571132768 +0000 UTC m=+2908.331158395" watchObservedRunningTime="2026-07-28 17:58:43.571676768 +0000 UTC m=+2908.331702396"
Jul 28 17:58:44.554013 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:44.553979    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:44.555182 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:44.555155    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:58:45.562783 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:45.562732    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:58:50.566965 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:50.566940    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:58:50.567496 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:58:50.567469    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:59:00.568118 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:00.568082    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:59:10.568055 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:10.568017    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:59:20.567626 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:20.567585    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:59:30.567973 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:30.567935    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:59:40.568038 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:40.567947    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:59:50.568305 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:50.568274    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:59:52.611833 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:52.611801    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"]
Jul 28 17:59:52.612206 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:52.612047    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container" containerID="cri-o://771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada" gracePeriod=30
Jul 28 17:59:52.612206 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:52.612090    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kube-rbac-proxy" containerID="cri-o://dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe" gracePeriod=30
Jul 28 17:59:52.779608 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:52.779581    2571 generic.go:356] "Generic (PLEG): container finished" podID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerID="dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe" exitCode=2
Jul 28 17:59:52.779704 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:52.779611    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerDied","Data":"dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe"}
Jul 28 17:59:55.339581 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.339558    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:59:55.404894 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.404837    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b018744-0116-4331-bd03-c0b897f3f7f3-proxy-tls\") pod \"4b018744-0116-4331-bd03-c0b897f3f7f3\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") "
Jul 28 17:59:55.404894 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.404877    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk57k\" (UniqueName: \"kubernetes.io/projected/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-api-access-xk57k\") pod \"4b018744-0116-4331-bd03-c0b897f3f7f3\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") "
Jul 28 17:59:55.405057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.404914    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-cabundle-cert\") pod \"4b018744-0116-4331-bd03-c0b897f3f7f3\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") "
Jul 28 17:59:55.405057 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.405029    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b018744-0116-4331-bd03-c0b897f3f7f3-kserve-provision-location\") pod \"4b018744-0116-4331-bd03-c0b897f3f7f3\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") "
Jul 28 17:59:55.405163 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.405109    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-rbac-proxy-sar-config\") pod \"4b018744-0116-4331-bd03-c0b897f3f7f3\" (UID: \"4b018744-0116-4331-bd03-c0b897f3f7f3\") "
Jul 28 17:59:55.405204 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.405160    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b018744-0116-4331-bd03-c0b897f3f7f3" (UID: "4b018744-0116-4331-bd03-c0b897f3f7f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:55.405360 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.405340    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.405455 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.405367    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b018744-0116-4331-bd03-c0b897f3f7f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b018744-0116-4331-bd03-c0b897f3f7f3" (UID: "4b018744-0116-4331-bd03-c0b897f3f7f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:55.405500 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.405451    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b018744-0116-4331-bd03-c0b897f3f7f3" (UID: "4b018744-0116-4331-bd03-c0b897f3f7f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:55.406881 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.406858    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-api-access-xk57k" (OuterVolumeSpecName: "kube-api-access-xk57k") pod "4b018744-0116-4331-bd03-c0b897f3f7f3" (UID: "4b018744-0116-4331-bd03-c0b897f3f7f3"). InnerVolumeSpecName "kube-api-access-xk57k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:55.406967 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.406862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b018744-0116-4331-bd03-c0b897f3f7f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b018744-0116-4331-bd03-c0b897f3f7f3" (UID: "4b018744-0116-4331-bd03-c0b897f3f7f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:55.506292 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.506271    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b018744-0116-4331-bd03-c0b897f3f7f3-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.506292 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.506290    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.506407 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.506299    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b018744-0116-4331-bd03-c0b897f3f7f3-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.506407 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.506310    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk57k\" (UniqueName: \"kubernetes.io/projected/4b018744-0116-4331-bd03-c0b897f3f7f3-kube-api-access-xk57k\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.794858 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.794832    2571 generic.go:356] "Generic (PLEG): container finished" podID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerID="771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada" exitCode=0
Jul 28 17:59:55.794955 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.794873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerDied","Data":"771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada"}
Jul 28 17:59:55.794955 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.794893    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"
Jul 28 17:59:55.794955 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.794924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb" event={"ID":"4b018744-0116-4331-bd03-c0b897f3f7f3","Type":"ContainerDied","Data":"b4d9b9c7b4bb8461fafebd4df104d9d7a70951c30221006181ead2545de0bfef"}
Jul 28 17:59:55.795060 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.794960    2571 scope.go:117] "RemoveContainer" containerID="dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe"
Jul 28 17:59:55.802293 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.802275    2571 scope.go:117] "RemoveContainer" containerID="771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada"
Jul 28 17:59:55.808903 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.808885    2571 scope.go:117] "RemoveContainer" containerID="a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794"
Jul 28 17:59:55.815183 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.815167    2571 scope.go:117] "RemoveContainer" containerID="dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe"
Jul 28 17:59:55.815461 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:59:55.815445    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe\": container with ID starting with dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe not found: ID does not exist" containerID="dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe"
Jul 28 17:59:55.815514 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.815467    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe"} err="failed to get container status \"dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe\": rpc error: code = NotFound desc = could not find container \"dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe\": container with ID starting with dacd45268a76a9c74a949c1bc38ca0d0c7908290bcfd83e5992bf226f1a21dbe not found: ID does not exist"
Jul 28 17:59:55.815514 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.815482    2571 scope.go:117] "RemoveContainer" containerID="771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada"
Jul 28 17:59:55.815683 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:59:55.815668    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada\": container with ID starting with 771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada not found: ID does not exist" containerID="771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada"
Jul 28 17:59:55.815725 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.815688    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada"} err="failed to get container status \"771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada\": rpc error: code = NotFound desc = could not find container \"771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada\": container with ID starting with 771102db3c05c2d54eb446634bce2fa9a76e9f5a8f25bddbf1eddb398e0d2ada not found: ID does not exist"
Jul 28 17:59:55.815725 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.815701    2571 scope.go:117] "RemoveContainer" containerID="a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794"
Jul 28 17:59:55.815977 ip-10-0-143-61 kubenswrapper[2571]: E0728 17:59:55.815950    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794\": container with ID starting with a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794 not found: ID does not exist" containerID="a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794"
Jul 28 17:59:55.816056 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.815984    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794"} err="failed to get container status \"a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794\": rpc error: code = NotFound desc = could not find container \"a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794\": container with ID starting with a3fa4c5d918d30f9f809412012635ee03752570281bb3013b6000131654ac794 not found: ID does not exist"
Jul 28 17:59:55.817878 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.817857    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"]
Jul 28 17:59:55.821963 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:55.821942    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-65rhb"]
Jul 28 17:59:57.706958 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.706917    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" path="/var/lib/kubelet/pods/4b018744-0116-4331-bd03-c0b897f3f7f3/volumes"
Jul 28 17:59:57.804445 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804417    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"]
Jul 28 17:59:57.804648 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804636    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="storage-initializer"
Jul 28 17:59:57.804688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804650    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="storage-initializer"
Jul 28 17:59:57.804688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804668    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kube-rbac-proxy"
Jul 28 17:59:57.804688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804673    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kube-rbac-proxy"
Jul 28 17:59:57.804688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804681    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container"
Jul 28 17:59:57.804688 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804687    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container"
Jul 28 17:59:57.804839 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804731    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kserve-container"
Jul 28 17:59:57.804839 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.804738    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b018744-0116-4331-bd03-c0b897f3f7f3" containerName="kube-rbac-proxy"
Jul 28 17:59:57.806910 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.806895    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:57.809673 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.809648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:59:57.809795 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.809669    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:59:57.809795 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.809699    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 17:59:57.809795 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.809716    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:59:57.809795 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.809652    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:59:57.809795 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.809652    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:59:57.810214 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.810199    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:59:57.818981 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.818963    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"]
Jul 28 17:59:57.920516 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.920497    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0555b333-3a73-4902-b79c-3fffcb1bea74-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:57.920614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.920523    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:57.920614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.920547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:57.920614 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.920573    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0555b333-3a73-4902-b79c-3fffcb1bea74-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:57.920731 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:57.920647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnvrd\" (UniqueName: \"kubernetes.io/projected/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-api-access-xnvrd\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.020922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.020869    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0555b333-3a73-4902-b79c-3fffcb1bea74-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.020922 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.020911    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnvrd\" (UniqueName: \"kubernetes.io/projected/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-api-access-xnvrd\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.021046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.020942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0555b333-3a73-4902-b79c-3fffcb1bea74-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.021046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.020959    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.021046 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.020987    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.021300 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.021284    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0555b333-3a73-4902-b79c-3fffcb1bea74-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.021710 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.021691    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.021746 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.021691    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.023391 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.023370    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0555b333-3a73-4902-b79c-3fffcb1bea74-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.031114 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.031093    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnvrd\" (UniqueName: \"kubernetes.io/projected/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-api-access-xnvrd\") pod \"isvc-pmml-runtime-predictor-885fc8777-xpm5g\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.116117 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.116097    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 17:59:58.242630 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.242584    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"]
Jul 28 17:59:58.244705 ip-10-0-143-61 kubenswrapper[2571]: W0728 17:59:58.244675    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0555b333_3a73_4902_b79c_3fffcb1bea74.slice/crio-69e9987eddb5039196014e816a539ccba955af651ab5e6e1766ccff63a40d174 WatchSource:0}: Error finding container 69e9987eddb5039196014e816a539ccba955af651ab5e6e1766ccff63a40d174: Status 404 returned error can't find the container with id 69e9987eddb5039196014e816a539ccba955af651ab5e6e1766ccff63a40d174
Jul 28 17:59:58.807639 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.807607    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerStarted","Data":"e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf"}
Jul 28 17:59:58.808006 ip-10-0-143-61 kubenswrapper[2571]: I0728 17:59:58.807662    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerStarted","Data":"69e9987eddb5039196014e816a539ccba955af651ab5e6e1766ccff63a40d174"}
Jul 28 18:00:02.822348 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:02.822319    2571 generic.go:356] "Generic (PLEG): container finished" podID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerID="e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf" exitCode=0
Jul 28 18:00:02.822645 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:02.822386    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerDied","Data":"e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf"}
Jul 28 18:00:03.826970 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:03.826938    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerStarted","Data":"75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798"}
Jul 28 18:00:03.827358 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:03.826990    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerStarted","Data":"4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f"}
Jul 28 18:00:03.827358 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:03.827270    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 18:00:03.827478 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:03.827410    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 18:00:03.828531 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:03.828510    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:03.854513 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:03.854468    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podStartSLOduration=3.012250209 podStartE2EDuration="6.854455373s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.842205164s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:03.8538848 +0000 UTC m=+2988.613910430" watchObservedRunningTime="2026-07-28 18:00:03.854455373 +0000 UTC m=+2988.614481002"
Jul 28 18:00:04.830515 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:04.830473    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:09.835334 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:09.835307    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 18:00:09.835901 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:09.835867    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:19.836786 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:19.836746    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:29.836574 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:29.836531    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:39.836170 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:39.836133    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:49.836686 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:49.836647    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:00:59.836680 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:00:59.836639    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:01:09.836458 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:09.836379    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 18:01:11.707208 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:11.707180    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 18:01:18.816452 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:18.816423    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"]
Jul 28 18:01:18.816834 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:18.816658    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kube-rbac-proxy" containerID="cri-o://75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798" gracePeriod=30
Jul 28 18:01:18.816834 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:18.816650    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container" containerID="cri-o://4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f" gracePeriod=30
Jul 28 18:01:19.074408 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:19.074348    2571 generic.go:356] "Generic (PLEG): container finished" podID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerID="75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798" exitCode=2
Jul 28 18:01:19.074507 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:19.074420    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerDied","Data":"75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798"}
Jul 28 18:01:19.830731 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:19.830695    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 28 18:01:21.346901 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.346881    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 18:01:21.456407 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456359    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnvrd\" (UniqueName: \"kubernetes.io/projected/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-api-access-xnvrd\") pod \"0555b333-3a73-4902-b79c-3fffcb1bea74\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") "
Jul 28 18:01:21.456407 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456395    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-rbac-proxy-sar-config\") pod \"0555b333-3a73-4902-b79c-3fffcb1bea74\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") "
Jul 28 18:01:21.456580 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456437    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0555b333-3a73-4902-b79c-3fffcb1bea74-proxy-tls\") pod \"0555b333-3a73-4902-b79c-3fffcb1bea74\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") "
Jul 28 18:01:21.456580 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456462    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-cabundle-cert\") pod \"0555b333-3a73-4902-b79c-3fffcb1bea74\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") "
Jul 28 18:01:21.456580 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456491    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0555b333-3a73-4902-b79c-3fffcb1bea74-kserve-provision-location\") pod \"0555b333-3a73-4902-b79c-3fffcb1bea74\" (UID: \"0555b333-3a73-4902-b79c-3fffcb1bea74\") "
Jul 28 18:01:21.456786 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456757    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0555b333-3a73-4902-b79c-3fffcb1bea74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0555b333-3a73-4902-b79c-3fffcb1bea74" (UID: "0555b333-3a73-4902-b79c-3fffcb1bea74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:01:21.456875 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456834    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0555b333-3a73-4902-b79c-3fffcb1bea74" (UID: "0555b333-3a73-4902-b79c-3fffcb1bea74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:21.456875 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.456849    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0555b333-3a73-4902-b79c-3fffcb1bea74" (UID: "0555b333-3a73-4902-b79c-3fffcb1bea74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:21.458315 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.458295    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0555b333-3a73-4902-b79c-3fffcb1bea74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0555b333-3a73-4902-b79c-3fffcb1bea74" (UID: "0555b333-3a73-4902-b79c-3fffcb1bea74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:01:21.458440 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.458425    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-api-access-xnvrd" (OuterVolumeSpecName: "kube-api-access-xnvrd") pod "0555b333-3a73-4902-b79c-3fffcb1bea74" (UID: "0555b333-3a73-4902-b79c-3fffcb1bea74"). InnerVolumeSpecName "kube-api-access-xnvrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:21.556956 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.556935    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0555b333-3a73-4902-b79c-3fffcb1bea74-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:01:21.556956 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.556957    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:01:21.557060 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.556966    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0555b333-3a73-4902-b79c-3fffcb1bea74-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:01:21.557060 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.556974    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnvrd\" (UniqueName: \"kubernetes.io/projected/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-api-access-xnvrd\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:01:21.557060 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:21.556983    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0555b333-3a73-4902-b79c-3fffcb1bea74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:01:22.085603 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.085547    2571 generic.go:356] "Generic (PLEG): container finished" podID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerID="4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f" exitCode=0
Jul 28 18:01:22.085603 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.085597    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"
Jul 28 18:01:22.085736 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.085624    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerDied","Data":"4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f"}
Jul 28 18:01:22.085736 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.085684    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g" event={"ID":"0555b333-3a73-4902-b79c-3fffcb1bea74","Type":"ContainerDied","Data":"69e9987eddb5039196014e816a539ccba955af651ab5e6e1766ccff63a40d174"}
Jul 28 18:01:22.085736 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.085722    2571 scope.go:117] "RemoveContainer" containerID="75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798"
Jul 28 18:01:22.093031 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.093014    2571 scope.go:117] "RemoveContainer" containerID="4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f"
Jul 28 18:01:22.099593 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.099571    2571 scope.go:117] "RemoveContainer" containerID="e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf"
Jul 28 18:01:22.105859 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.105837    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"]
Jul 28 18:01:22.106868 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.106852    2571 scope.go:117] "RemoveContainer" containerID="75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798"
Jul 28 18:01:22.107123 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:01:22.107107    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798\": container with ID starting with 75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798 not found: ID does not exist" containerID="75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798"
Jul 28 18:01:22.107169 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.107130    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798"} err="failed to get container status \"75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798\": rpc error: code = NotFound desc = could not find container \"75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798\": container with ID starting with 75dddb9da6c677e7ad78b5198a95c18f647247654417d32a39637013d6b71798 not found: ID does not exist"
Jul 28 18:01:22.107169 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.107146    2571 scope.go:117] "RemoveContainer" containerID="4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f"
Jul 28 18:01:22.107400 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:01:22.107380    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f\": container with ID starting with 4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f not found: ID does not exist" containerID="4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f"
Jul 28 18:01:22.107518 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.107405    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f"} err="failed to get container status \"4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f\": rpc error: code = NotFound desc = could not find container \"4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f\": container with ID starting with 4fca4f7a620becde2ecbdde2438e15abf660baf82d8aa368f87a4810cd76682f not found: ID does not exist"
Jul 28 18:01:22.107518 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.107427    2571 scope.go:117] "RemoveContainer" containerID="e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf"
Jul 28 18:01:22.107742 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:01:22.107670    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf\": container with ID starting with e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf not found: ID does not exist" containerID="e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf"
Jul 28 18:01:22.107742 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.107701    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf"} err="failed to get container status \"e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf\": rpc error: code = NotFound desc = could not find container \"e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf\": container with ID starting with e7bdfe9d7fd7816945db446eaf3766154515fac61731224ca375509259bd97cf not found: ID does not exist"
Jul 28 18:01:22.109224 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:22.109206    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-xpm5g"]
Jul 28 18:01:23.707377 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.707344    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" path="/var/lib/kubelet/pods/0555b333-3a73-4902-b79c-3fffcb1bea74/volumes"
Jul 28 18:01:23.986858 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.986792    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"]
Jul 28 18:01:23.987028 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987017    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="storage-initializer"
Jul 28 18:01:23.987079 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987029    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="storage-initializer"
Jul 28 18:01:23.987079 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987038    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kube-rbac-proxy"
Jul 28 18:01:23.987079 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987043    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kube-rbac-proxy"
Jul 28 18:01:23.987079 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987055    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container"
Jul 28 18:01:23.987079 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987061    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container"
Jul 28 18:01:23.987256 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987100    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kserve-container"
Jul 28 18:01:23.987256 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.987110    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0555b333-3a73-4902-b79c-3fffcb1bea74" containerName="kube-rbac-proxy"
Jul 28 18:01:23.991313 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.991298    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:23.994353 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.994331    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:01:23.994479 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.994410    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:01:23.994602 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.994586    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 18:01:23.995284 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.995259    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 18:01:23.995284 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.995265    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:01:23.995535 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.995520    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:01:23.995616 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:23.995580    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:01:24.003902 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.003879    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"]
Jul 28 18:01:24.173446 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.173418    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/148813ee-4c81-4bac-8085-a7c6396a804b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.173544 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.173454    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zn6\" (UniqueName: \"kubernetes.io/projected/148813ee-4c81-4bac-8085-a7c6396a804b-kube-api-access-76zn6\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.173544 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.173477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.173627 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.173548    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.173668 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.173632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148813ee-4c81-4bac-8085-a7c6396a804b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.274760 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.274712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.274760 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.274754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148813ee-4c81-4bac-8085-a7c6396a804b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.274894 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.274772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/148813ee-4c81-4bac-8085-a7c6396a804b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.274894 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.274790    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zn6\" (UniqueName: \"kubernetes.io/projected/148813ee-4c81-4bac-8085-a7c6396a804b-kube-api-access-76zn6\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.274987 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.274906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.275117 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.275099    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/148813ee-4c81-4bac-8085-a7c6396a804b-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.275397 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.275381    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.275474 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.275459    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.276967 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.276947    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148813ee-4c81-4bac-8085-a7c6396a804b-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.284167 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.284149    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zn6\" (UniqueName: \"kubernetes.io/projected/148813ee-4c81-4bac-8085-a7c6396a804b-kube-api-access-76zn6\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.300059 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.300040    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:24.426036 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:24.426004    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"]
Jul 28 18:01:24.429026 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:01:24.429000    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148813ee_4c81_4bac_8085_a7c6396a804b.slice/crio-11937468dc6ab972837accb1fd50976b167a1fd55d4f287fa5d871114fee341b WatchSource:0}: Error finding container 11937468dc6ab972837accb1fd50976b167a1fd55d4f287fa5d871114fee341b: Status 404 returned error can't find the container with id 11937468dc6ab972837accb1fd50976b167a1fd55d4f287fa5d871114fee341b
Jul 28 18:01:25.097823 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:25.097790    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerStarted","Data":"16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a"}
Jul 28 18:01:25.098193 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:25.097854    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerStarted","Data":"11937468dc6ab972837accb1fd50976b167a1fd55d4f287fa5d871114fee341b"}
Jul 28 18:01:29.114524 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:29.114496    2571 generic.go:356] "Generic (PLEG): container finished" podID="148813ee-4c81-4bac-8085-a7c6396a804b" containerID="16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a" exitCode=0
Jul 28 18:01:29.114850 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:29.114566    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerDied","Data":"16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a"}
Jul 28 18:01:30.119164 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:30.119132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerStarted","Data":"fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e"}
Jul 28 18:01:30.119598 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:30.119192    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerStarted","Data":"22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4"}
Jul 28 18:01:30.119598 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:30.119413    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:30.146877 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:30.146827    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podStartSLOduration=3.396014469 podStartE2EDuration="7.146811311s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750796842s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:30.144296197 +0000 UTC m=+3074.904321825" watchObservedRunningTime="2026-07-28 18:01:30.146811311 +0000 UTC m=+3074.906836944"
Jul 28 18:01:31.122435 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:31.122409    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:31.123532 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:31.123497    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:01:32.125674 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:32.125637    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:01:37.130197 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:37.130168    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:01:37.130648 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:37.130622    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:01:47.131632 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:47.131598    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:01:57.130882 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:01:57.130844    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:02:07.131000 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:07.130965    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:02:17.131089 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:17.131053    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:02:27.131317 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:27.131274    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:02:37.131668 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:37.131641    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:02:44.918461 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:44.918430    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"]
Jul 28 18:02:44.918832 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:44.918649    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" containerID="cri-o://22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4" gracePeriod=30
Jul 28 18:02:44.918832 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:44.918672    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kube-rbac-proxy" containerID="cri-o://fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e" gracePeriod=30
Jul 28 18:02:45.363786 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:45.363721    2571 generic.go:356] "Generic (PLEG): container finished" podID="148813ee-4c81-4bac-8085-a7c6396a804b" containerID="fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e" exitCode=2
Jul 28 18:02:45.363786 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:45.363770    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerDied","Data":"fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e"}
Jul 28 18:02:47.125905 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.125872    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 28 18:02:47.131296 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.131271    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 18:02:47.656084 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.656062    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:02:47.716720 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.716653    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/148813ee-4c81-4bac-8085-a7c6396a804b-kserve-provision-location\") pod \"148813ee-4c81-4bac-8085-a7c6396a804b\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") "
Jul 28 18:02:47.716720 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.716705    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-cabundle-cert\") pod \"148813ee-4c81-4bac-8085-a7c6396a804b\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") "
Jul 28 18:02:47.716871 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.716738    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76zn6\" (UniqueName: \"kubernetes.io/projected/148813ee-4c81-4bac-8085-a7c6396a804b-kube-api-access-76zn6\") pod \"148813ee-4c81-4bac-8085-a7c6396a804b\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") "
Jul 28 18:02:47.716871 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.716768    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148813ee-4c81-4bac-8085-a7c6396a804b-proxy-tls\") pod \"148813ee-4c81-4bac-8085-a7c6396a804b\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") "
Jul 28 18:02:47.716871 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.716835    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-kube-rbac-proxy-sar-config\") pod \"148813ee-4c81-4bac-8085-a7c6396a804b\" (UID: \"148813ee-4c81-4bac-8085-a7c6396a804b\") "
Jul 28 18:02:47.717034 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.716949    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148813ee-4c81-4bac-8085-a7c6396a804b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "148813ee-4c81-4bac-8085-a7c6396a804b" (UID: "148813ee-4c81-4bac-8085-a7c6396a804b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:47.717034 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.717004    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/148813ee-4c81-4bac-8085-a7c6396a804b-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:02:47.717142 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.717036    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "148813ee-4c81-4bac-8085-a7c6396a804b" (UID: "148813ee-4c81-4bac-8085-a7c6396a804b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:02:47.717234 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.717216    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "148813ee-4c81-4bac-8085-a7c6396a804b" (UID: "148813ee-4c81-4bac-8085-a7c6396a804b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:02:47.718753 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.718733    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148813ee-4c81-4bac-8085-a7c6396a804b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "148813ee-4c81-4bac-8085-a7c6396a804b" (UID: "148813ee-4c81-4bac-8085-a7c6396a804b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:47.718866 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.718845    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148813ee-4c81-4bac-8085-a7c6396a804b-kube-api-access-76zn6" (OuterVolumeSpecName: "kube-api-access-76zn6") pod "148813ee-4c81-4bac-8085-a7c6396a804b" (UID: "148813ee-4c81-4bac-8085-a7c6396a804b"). InnerVolumeSpecName "kube-api-access-76zn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:47.818164 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.818142    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:02:47.818164 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.818162    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/148813ee-4c81-4bac-8085-a7c6396a804b-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:02:47.818312 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.818171    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zn6\" (UniqueName: \"kubernetes.io/projected/148813ee-4c81-4bac-8085-a7c6396a804b-kube-api-access-76zn6\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:02:47.818312 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:47.818180    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/148813ee-4c81-4bac-8085-a7c6396a804b-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:02:48.375085 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.375062    2571 generic.go:356] "Generic (PLEG): container finished" podID="148813ee-4c81-4bac-8085-a7c6396a804b" containerID="22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4" exitCode=0
Jul 28 18:02:48.375424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.375113    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"
Jul 28 18:02:48.375424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.375144    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerDied","Data":"22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4"}
Jul 28 18:02:48.375424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.375199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2" event={"ID":"148813ee-4c81-4bac-8085-a7c6396a804b","Type":"ContainerDied","Data":"11937468dc6ab972837accb1fd50976b167a1fd55d4f287fa5d871114fee341b"}
Jul 28 18:02:48.375424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.375226    2571 scope.go:117] "RemoveContainer" containerID="fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e"
Jul 28 18:02:48.383071 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.382944    2571 scope.go:117] "RemoveContainer" containerID="22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4"
Jul 28 18:02:48.389782 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.389767    2571 scope.go:117] "RemoveContainer" containerID="16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a"
Jul 28 18:02:48.396236 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.396220    2571 scope.go:117] "RemoveContainer" containerID="fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e"
Jul 28 18:02:48.396507 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:02:48.396490    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e\": container with ID starting with fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e not found: ID does not exist" containerID="fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e"
Jul 28 18:02:48.396569 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.396512    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e"} err="failed to get container status \"fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e\": rpc error: code = NotFound desc = could not find container \"fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e\": container with ID starting with fe47234fe4c9b09794b3a4dffaeae6d372e30929bd9341deacbc281054ce0e5e not found: ID does not exist"
Jul 28 18:02:48.396569 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.396527    2571 scope.go:117] "RemoveContainer" containerID="22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4"
Jul 28 18:02:48.396756 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:02:48.396739    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4\": container with ID starting with 22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4 not found: ID does not exist" containerID="22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4"
Jul 28 18:02:48.396796 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.396759    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4"} err="failed to get container status \"22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4\": rpc error: code = NotFound desc = could not find container \"22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4\": container with ID starting with 22c19583a490f7fcb4244613bdbcddc19e24b593621269926d00cac6b9c0f9e4 not found: ID does not exist"
Jul 28 18:02:48.396796 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.396772    2571 scope.go:117] "RemoveContainer" containerID="16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a"
Jul 28 18:02:48.396988 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:02:48.396971    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a\": container with ID starting with 16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a not found: ID does not exist" containerID="16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a"
Jul 28 18:02:48.397041 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.396994    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a"} err="failed to get container status \"16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a\": rpc error: code = NotFound desc = could not find container \"16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a\": container with ID starting with 16afdd395dadd556f737c5ae02a0bb81b2ac82cf3b324f8d2d49a4d8c569894a not found: ID does not exist"
Jul 28 18:02:48.397041 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.397001    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"]
Jul 28 18:02:48.400929 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:48.400912    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-s6qs2"]
Jul 28 18:02:49.711040 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:49.711005    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" path="/var/lib/kubelet/pods/148813ee-4c81-4bac-8085-a7c6396a804b/volumes"
Jul 28 18:02:50.104742 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104674    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"]
Jul 28 18:02:50.104905 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104893    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="storage-initializer"
Jul 28 18:02:50.104951 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104906    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="storage-initializer"
Jul 28 18:02:50.104951 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104919    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container"
Jul 28 18:02:50.104951 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104924    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container"
Jul 28 18:02:50.104951 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104930    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kube-rbac-proxy"
Jul 28 18:02:50.104951 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104935    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kube-rbac-proxy"
Jul 28 18:02:50.105096 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104970    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kserve-container"
Jul 28 18:02:50.105096 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.104978    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="148813ee-4c81-4bac-8085-a7c6396a804b" containerName="kube-rbac-proxy"
Jul 28 18:02:50.109359 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.109344    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.112066 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.112040    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-de1b83-predictor-serving-cert\""
Jul 28 18:02:50.112066 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.112057    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:02:50.112260 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.112138    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:02:50.112260 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.112161    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:02:50.113221 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.113200    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:02:50.113363 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.113202    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-de1b83-kube-rbac-proxy-sar-config\""
Jul 28 18:02:50.113363 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.113265    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:02:50.123912 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.123893    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"]
Jul 28 18:02:50.234963 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.234943    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-cabundle-cert\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.235088 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.234983    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c475b821-2a26-4d6c-8cd4-f82062c5d355-kserve-provision-location\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.235088 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.235012    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.235088 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.235076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbfx\" (UniqueName: \"kubernetes.io/projected/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-api-access-bdbfx\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.235218 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.235130    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-rbac-proxy-sar-config\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.335674 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.335644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.335674 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.335679    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbfx\" (UniqueName: \"kubernetes.io/projected/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-api-access-bdbfx\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.335793 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.335700    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-rbac-proxy-sar-config\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.335793 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.335724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-cabundle-cert\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.335793 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.335773    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c475b821-2a26-4d6c-8cd4-f82062c5d355-kserve-provision-location\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.335911 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:02:50.335795    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-de1b83-predictor-serving-cert: secret "isvc-primary-de1b83-predictor-serving-cert" not found
Jul 28 18:02:50.335911 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:02:50.335870    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls podName:c475b821-2a26-4d6c-8cd4-f82062c5d355 nodeName:}" failed. No retries permitted until 2026-07-28 18:02:50.835830311 +0000 UTC m=+3155.595855921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls") pod "isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" (UID: "c475b821-2a26-4d6c-8cd4-f82062c5d355") : secret "isvc-primary-de1b83-predictor-serving-cert" not found
Jul 28 18:02:50.336128 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.336112    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c475b821-2a26-4d6c-8cd4-f82062c5d355-kserve-provision-location\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.336306 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.336290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-rbac-proxy-sar-config\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.336413 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.336400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-cabundle-cert\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.347599 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.347581    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbfx\" (UniqueName: \"kubernetes.io/projected/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-api-access-bdbfx\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.839609 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.839587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:50.841708 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:50.841691    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls\") pod \"isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") " pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:51.018448 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:51.018418    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:51.142010 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:51.141982    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"]
Jul 28 18:02:51.145010 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:02:51.144984    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc475b821_2a26_4d6c_8cd4_f82062c5d355.slice/crio-07d6c13d3e710f908f612a5f60ea219441cb3992ed8a0c4fcb3cb019c265e67f WatchSource:0}: Error finding container 07d6c13d3e710f908f612a5f60ea219441cb3992ed8a0c4fcb3cb019c265e67f: Status 404 returned error can't find the container with id 07d6c13d3e710f908f612a5f60ea219441cb3992ed8a0c4fcb3cb019c265e67f
Jul 28 18:02:51.388332 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:51.388305    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerStarted","Data":"ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5"}
Jul 28 18:02:51.388452 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:51.388361    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerStarted","Data":"07d6c13d3e710f908f612a5f60ea219441cb3992ed8a0c4fcb3cb019c265e67f"}
Jul 28 18:02:55.401873 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:55.401843    2571 generic.go:356] "Generic (PLEG): container finished" podID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerID="ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5" exitCode=0
Jul 28 18:02:55.402178 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:55.401913    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerDied","Data":"ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5"}
Jul 28 18:02:56.406391 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:56.406362    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerStarted","Data":"f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b"}
Jul 28 18:02:56.406789 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:56.406440    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerStarted","Data":"0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0"}
Jul 28 18:02:56.406789 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:56.406628    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:56.406789 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:56.406672    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:02:56.407925 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:56.407907    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:02:56.428923 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:56.428884    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podStartSLOduration=2.767000328 podStartE2EDuration="6.428870983s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.661870655s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:56.426764186 +0000 UTC m=+3161.186789824" watchObservedRunningTime="2026-07-28 18:02:56.428870983 +0000 UTC m=+3161.188896611"
Jul 28 18:02:57.410291 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:02:57.410240    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:03:02.414210 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:02.414184    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:03:02.414693 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:02.414668    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:03:12.415309 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:12.415268    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:03:22.414908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:22.414866    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:03:32.414710 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:32.414670    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:03:42.414778 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:42.414738    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:03:52.415277 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:03:52.415224    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:04:02.415528 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:02.415488    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 28 18:04:09.706422 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:09.706355    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:04:43.485095 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:43.485062    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"]
Jul 28 18:04:43.485612 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:43.485344    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container" containerID="cri-o://0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0" gracePeriod=30
Jul 28 18:04:43.485612 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:43.485411    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kube-rbac-proxy" containerID="cri-o://f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b" gracePeriod=30
Jul 28 18:04:43.773426 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:43.773364    2571 generic.go:356] "Generic (PLEG): container finished" podID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerID="f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b" exitCode=2
Jul 28 18:04:43.773426 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:43.773410    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerDied","Data":"f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b"}
Jul 28 18:04:47.129343 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.129322    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:04:47.234747 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.234688    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c475b821-2a26-4d6c-8cd4-f82062c5d355-kserve-provision-location\") pod \"c475b821-2a26-4d6c-8cd4-f82062c5d355\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") "
Jul 28 18:04:47.234747 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.234721    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-cabundle-cert\") pod \"c475b821-2a26-4d6c-8cd4-f82062c5d355\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") "
Jul 28 18:04:47.234893 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.234749    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls\") pod \"c475b821-2a26-4d6c-8cd4-f82062c5d355\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") "
Jul 28 18:04:47.234893 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.234874    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-rbac-proxy-sar-config\") pod \"c475b821-2a26-4d6c-8cd4-f82062c5d355\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") "
Jul 28 18:04:47.234973 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.234914    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdbfx\" (UniqueName: \"kubernetes.io/projected/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-api-access-bdbfx\") pod \"c475b821-2a26-4d6c-8cd4-f82062c5d355\" (UID: \"c475b821-2a26-4d6c-8cd4-f82062c5d355\") "
Jul 28 18:04:47.235150 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.235120    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c475b821-2a26-4d6c-8cd4-f82062c5d355-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c475b821-2a26-4d6c-8cd4-f82062c5d355" (UID: "c475b821-2a26-4d6c-8cd4-f82062c5d355"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:47.235289 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.235162    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c475b821-2a26-4d6c-8cd4-f82062c5d355" (UID: "c475b821-2a26-4d6c-8cd4-f82062c5d355"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:47.235289 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.235204    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c475b821-2a26-4d6c-8cd4-f82062c5d355" (UID: "c475b821-2a26-4d6c-8cd4-f82062c5d355"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:47.236808 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.236786    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c475b821-2a26-4d6c-8cd4-f82062c5d355" (UID: "c475b821-2a26-4d6c-8cd4-f82062c5d355"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:47.236899 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.236813    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-api-access-bdbfx" (OuterVolumeSpecName: "kube-api-access-bdbfx") pod "c475b821-2a26-4d6c-8cd4-f82062c5d355" (UID: "c475b821-2a26-4d6c-8cd4-f82062c5d355"). InnerVolumeSpecName "kube-api-access-bdbfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:47.335826 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.335806    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:47.335826 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.335827    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdbfx\" (UniqueName: \"kubernetes.io/projected/c475b821-2a26-4d6c-8cd4-f82062c5d355-kube-api-access-bdbfx\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:47.335964 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.335841    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c475b821-2a26-4d6c-8cd4-f82062c5d355-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:47.335964 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.335853    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c475b821-2a26-4d6c-8cd4-f82062c5d355-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:47.335964 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.335867    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c475b821-2a26-4d6c-8cd4-f82062c5d355-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:47.792215 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.792187    2571 generic.go:356] "Generic (PLEG): container finished" podID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerID="0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0" exitCode=0
Jul 28 18:04:47.792344 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.792235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerDied","Data":"0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0"}
Jul 28 18:04:47.792344 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.792321    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"
Jul 28 18:04:47.792344 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.792336    2571 scope.go:117] "RemoveContainer" containerID="f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b"
Jul 28 18:04:47.792453 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.792322    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg" event={"ID":"c475b821-2a26-4d6c-8cd4-f82062c5d355","Type":"ContainerDied","Data":"07d6c13d3e710f908f612a5f60ea219441cb3992ed8a0c4fcb3cb019c265e67f"}
Jul 28 18:04:47.799846 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.799829    2571 scope.go:117] "RemoveContainer" containerID="0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0"
Jul 28 18:04:47.806323 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.806308    2571 scope.go:117] "RemoveContainer" containerID="ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5"
Jul 28 18:04:47.810972 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.810952    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"]
Jul 28 18:04:47.813814 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.813797    2571 scope.go:117] "RemoveContainer" containerID="f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b"
Jul 28 18:04:47.814054 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:04:47.814035    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b\": container with ID starting with f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b not found: ID does not exist" containerID="f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b"
Jul 28 18:04:47.814143 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.814064    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b"} err="failed to get container status \"f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b\": rpc error: code = NotFound desc = could not find container \"f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b\": container with ID starting with f715c559fcf0e75db4737dcc9683e94395438b8127576bea89f5ba3d2fd0118b not found: ID does not exist"
Jul 28 18:04:47.814143 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.814086    2571 scope.go:117] "RemoveContainer" containerID="0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0"
Jul 28 18:04:47.814237 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.814161    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-de1b83-predictor-dfb6f95d9-lv8rg"]
Jul 28 18:04:47.814349 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:04:47.814330    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0\": container with ID starting with 0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0 not found: ID does not exist" containerID="0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0"
Jul 28 18:04:47.814411 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.814359    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0"} err="failed to get container status \"0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0\": rpc error: code = NotFound desc = could not find container \"0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0\": container with ID starting with 0a2d6c69a0afa45cae0d411bfd0b9987691ff70629778ba5ae3b83f571baa6f0 not found: ID does not exist"
Jul 28 18:04:47.814411 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.814381    2571 scope.go:117] "RemoveContainer" containerID="ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5"
Jul 28 18:04:47.814633 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:04:47.814616    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5\": container with ID starting with ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5 not found: ID does not exist" containerID="ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5"
Jul 28 18:04:47.814676 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:47.814637    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5"} err="failed to get container status \"ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5\": rpc error: code = NotFound desc = could not find container \"ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5\": container with ID starting with ffaf52c42292e63e26c6ce818d021ffb4c7e1c1a2b3853fbb155becd4d2bb8e5 not found: ID does not exist"
Jul 28 18:04:48.689931 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.689902    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"]
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690129    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="storage-initializer"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690140    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="storage-initializer"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690152    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kube-rbac-proxy"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690158    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kube-rbac-proxy"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690168    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690174    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690211    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kserve-container"
Jul 28 18:04:48.690286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.690218    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" containerName="kube-rbac-proxy"
Jul 28 18:04:48.694355 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.694334    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.697089 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.697060    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-168fb5-predictor-serving-cert\""
Jul 28 18:04:48.697089 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.697082    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-168fb5-dockercfg-5b4m2\""
Jul 28 18:04:48.697089 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.697065    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-168fb5\""
Jul 28 18:04:48.697388 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.697137    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-168fb5-kube-rbac-proxy-sar-config\""
Jul 28 18:04:48.697436 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.697427    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:04:48.698321 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.698301    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:04:48.698421 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.698335    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:04:48.704579 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.704553    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"]
Jul 28 18:04:48.747353 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.747323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046fdd75-bacf-420b-80e4-1b82b8efa549-proxy-tls\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.747466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.747358    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.747466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.747389    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jk24\" (UniqueName: \"kubernetes.io/projected/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-api-access-4jk24\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.747466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.747405    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046fdd75-bacf-420b-80e4-1b82b8efa549-kserve-provision-location\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.747621 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.747511    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-cabundle-cert\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848011 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.847979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-cabundle-cert\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848029    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046fdd75-bacf-420b-80e4-1b82b8efa549-proxy-tls\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jk24\" (UniqueName: \"kubernetes.io/projected/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-api-access-4jk24\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046fdd75-bacf-420b-80e4-1b82b8efa549-kserve-provision-location\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848702 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848674    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046fdd75-bacf-420b-80e4-1b82b8efa549-kserve-provision-location\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848802 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848724    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.848802 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.848772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-cabundle-cert\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.850590 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.850571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046fdd75-bacf-420b-80e4-1b82b8efa549-proxy-tls\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:48.857414 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:48.857396    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jk24\" (UniqueName: \"kubernetes.io/projected/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-api-access-4jk24\") pod \"isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") " pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:49.004566 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:49.004516    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:49.128344 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:49.128308    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"]
Jul 28 18:04:49.131739 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:04:49.131704    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046fdd75_bacf_420b_80e4_1b82b8efa549.slice/crio-4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8 WatchSource:0}: Error finding container 4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8: Status 404 returned error can't find the container with id 4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8
Jul 28 18:04:49.706823 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:49.706792    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c475b821-2a26-4d6c-8cd4-f82062c5d355" path="/var/lib/kubelet/pods/c475b821-2a26-4d6c-8cd4-f82062c5d355/volumes"
Jul 28 18:04:49.801156 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:49.801125    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" event={"ID":"046fdd75-bacf-420b-80e4-1b82b8efa549","Type":"ContainerStarted","Data":"f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f"}
Jul 28 18:04:49.801323 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:49.801181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" event={"ID":"046fdd75-bacf-420b-80e4-1b82b8efa549","Type":"ContainerStarted","Data":"4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8"}
Jul 28 18:04:53.814402 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:53.814378    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/0.log"
Jul 28 18:04:53.814764 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:53.814414    2571 generic.go:356] "Generic (PLEG): container finished" podID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerID="f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f" exitCode=1
Jul 28 18:04:53.814764 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:53.814492    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" event={"ID":"046fdd75-bacf-420b-80e4-1b82b8efa549","Type":"ContainerDied","Data":"f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f"}
Jul 28 18:04:54.819777 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:54.819753    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/0.log"
Jul 28 18:04:54.820144 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:54.819834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" event={"ID":"046fdd75-bacf-420b-80e4-1b82b8efa549","Type":"ContainerStarted","Data":"3bcb5dcc8a09f127b521cf76ce32d6fd8166d7515ce274be8b78103e53f698da"}
Jul 28 18:04:56.828878 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.828855    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/1.log"
Jul 28 18:04:56.829206 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.829194    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/0.log"
Jul 28 18:04:56.829270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.829226    2571 generic.go:356] "Generic (PLEG): container finished" podID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerID="3bcb5dcc8a09f127b521cf76ce32d6fd8166d7515ce274be8b78103e53f698da" exitCode=1
Jul 28 18:04:56.829326 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.829306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" event={"ID":"046fdd75-bacf-420b-80e4-1b82b8efa549","Type":"ContainerDied","Data":"3bcb5dcc8a09f127b521cf76ce32d6fd8166d7515ce274be8b78103e53f698da"}
Jul 28 18:04:56.829396 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.829375    2571 scope.go:117] "RemoveContainer" containerID="f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f"
Jul 28 18:04:56.829675 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.829657    2571 scope.go:117] "RemoveContainer" containerID="f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f"
Jul 28 18:04:56.839532 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:04:56.839510    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_kserve-ci-e2e-test(046fdd75-bacf-420b-80e4-1b82b8efa549)\"" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549"
Jul 28 18:04:56.839867 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:04:56.839828    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_kserve-ci-e2e-test_046fdd75-bacf-420b-80e4-1b82b8efa549_0 in pod sandbox 4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8 from index: no such id: 'f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f'" containerID="f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f"
Jul 28 18:04:56.839946 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:56.839873    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_kserve-ci-e2e-test_046fdd75-bacf-420b-80e4-1b82b8efa549_0 in pod sandbox 4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8 from index: no such id: 'f5c6694047a80f83897872745c2ac16fdd4807ff521fffc5122a6ff549e7617f'"
Jul 28 18:04:57.833147 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:57.833127    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/1.log"
Jul 28 18:04:58.674927 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.674897    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"]
Jul 28 18:04:58.807529 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.807509    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/1.log"
Jul 28 18:04:58.811042 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.811028    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:58.837649 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.837628    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6_046fdd75-bacf-420b-80e4-1b82b8efa549/storage-initializer/1.log"
Jul 28 18:04:58.837952 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.837705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6" event={"ID":"046fdd75-bacf-420b-80e4-1b82b8efa549","Type":"ContainerDied","Data":"4143a3509bb54ba1f5f952620464543935e967c0dc58da761e0a93a8d392f1a8"}
Jul 28 18:04:58.837952 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.837735    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"
Jul 28 18:04:58.837952 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.837768    2571 scope.go:117] "RemoveContainer" containerID="3bcb5dcc8a09f127b521cf76ce32d6fd8166d7515ce274be8b78103e53f698da"
Jul 28 18:04:58.911692 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.911672    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046fdd75-bacf-420b-80e4-1b82b8efa549-kserve-provision-location\") pod \"046fdd75-bacf-420b-80e4-1b82b8efa549\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") "
Jul 28 18:04:58.911768 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.911713    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jk24\" (UniqueName: \"kubernetes.io/projected/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-api-access-4jk24\") pod \"046fdd75-bacf-420b-80e4-1b82b8efa549\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") "
Jul 28 18:04:58.911768 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.911754    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-cabundle-cert\") pod \"046fdd75-bacf-420b-80e4-1b82b8efa549\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") "
Jul 28 18:04:58.911861 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.911777    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046fdd75-bacf-420b-80e4-1b82b8efa549-proxy-tls\") pod \"046fdd75-bacf-420b-80e4-1b82b8efa549\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") "
Jul 28 18:04:58.911861 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.911794    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-rbac-proxy-sar-config\") pod \"046fdd75-bacf-420b-80e4-1b82b8efa549\" (UID: \"046fdd75-bacf-420b-80e4-1b82b8efa549\") "
Jul 28 18:04:58.911966 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.911907    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046fdd75-bacf-420b-80e4-1b82b8efa549-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "046fdd75-bacf-420b-80e4-1b82b8efa549" (UID: "046fdd75-bacf-420b-80e4-1b82b8efa549"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:58.912128 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.912109    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "046fdd75-bacf-420b-80e4-1b82b8efa549" (UID: "046fdd75-bacf-420b-80e4-1b82b8efa549"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:58.912187 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.912143    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "046fdd75-bacf-420b-80e4-1b82b8efa549" (UID: "046fdd75-bacf-420b-80e4-1b82b8efa549"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:58.913718 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.913698    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046fdd75-bacf-420b-80e4-1b82b8efa549-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "046fdd75-bacf-420b-80e4-1b82b8efa549" (UID: "046fdd75-bacf-420b-80e4-1b82b8efa549"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:58.913718 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:58.913702    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-api-access-4jk24" (OuterVolumeSpecName: "kube-api-access-4jk24") pod "046fdd75-bacf-420b-80e4-1b82b8efa549" (UID: "046fdd75-bacf-420b-80e4-1b82b8efa549"). InnerVolumeSpecName "kube-api-access-4jk24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:59.013023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.012974    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:59.013023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.012996    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/046fdd75-bacf-420b-80e4-1b82b8efa549-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:59.013023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.013006    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:59.013023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.013014    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/046fdd75-bacf-420b-80e4-1b82b8efa549-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:59.013023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.013023    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jk24\" (UniqueName: \"kubernetes.io/projected/046fdd75-bacf-420b-80e4-1b82b8efa549-kube-api-access-4jk24\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:04:59.179494 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.179470    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"]
Jul 28 18:04:59.185074 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.185054    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-168fb5-predictor-77fc576f84-hdgt6"]
Jul 28 18:04:59.707261 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:04:59.707229    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" path="/var/lib/kubelet/pods/046fdd75-bacf-420b-80e4-1b82b8efa549/volumes"
Jul 28 18:05:03.904876 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.904842    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"]
Jul 28 18:05:03.905230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.905086    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerName="storage-initializer"
Jul 28 18:05:03.905230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.905097    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerName="storage-initializer"
Jul 28 18:05:03.905230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.905150    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerName="storage-initializer"
Jul 28 18:05:03.905230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.905201    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerName="storage-initializer"
Jul 28 18:05:03.905230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.905207    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerName="storage-initializer"
Jul 28 18:05:03.905435 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.905257    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="046fdd75-bacf-420b-80e4-1b82b8efa549" containerName="storage-initializer"
Jul 28 18:05:03.911614 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.911589    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:03.916572 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.916529    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 18:05:03.916778 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.916534    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:05:03.916778 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.916642    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:05:03.916778 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.916581    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 18:05:03.916778 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.916695    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:05:03.916778 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.916546    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:05:03.919657 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.919630    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:05:03.921289 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:03.921270    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"]
Jul 28 18:05:04.045564 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.045535    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.045694 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.045572    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0fc5db0-4cfd-4830-937b-98e01be2031d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.045694 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.045613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.045694 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.045645    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmrk\" (UniqueName: \"kubernetes.io/projected/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-api-access-8pmrk\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.045694 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.045679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.146671 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.146651    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.146759 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.146678    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0fc5db0-4cfd-4830-937b-98e01be2031d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.146759 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.146703    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.146759 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.146736    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmrk\" (UniqueName: \"kubernetes.io/projected/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-api-access-8pmrk\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.146759 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.146756    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.147010 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:05:04.146940    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 28 18:05:04.147055 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:05:04.147010    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls podName:f0fc5db0-4cfd-4830-937b-98e01be2031d nodeName:}" failed. No retries permitted until 2026-07-28 18:05:04.646991731 +0000 UTC m=+3289.407017358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" (UID: "f0fc5db0-4cfd-4830-937b-98e01be2031d") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 28 18:05:04.147141 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.147120    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0fc5db0-4cfd-4830-937b-98e01be2031d-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.147431 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.147414    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.147510 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.147461    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.155599 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.155556    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmrk\" (UniqueName: \"kubernetes.io/projected/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-api-access-8pmrk\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.650807 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.650780    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.652857 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.652842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hl6gs\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.821960 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.821934    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:04.947676 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:04.947604    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"]
Jul 28 18:05:04.950621 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:05:04.950596    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fc5db0_4cfd_4830_937b_98e01be2031d.slice/crio-6d9c0a8a5f9d6c68de1295e210482551c9508d9af9551e57c978f18eb891e467 WatchSource:0}: Error finding container 6d9c0a8a5f9d6c68de1295e210482551c9508d9af9551e57c978f18eb891e467: Status 404 returned error can't find the container with id 6d9c0a8a5f9d6c68de1295e210482551c9508d9af9551e57c978f18eb891e467
Jul 28 18:05:05.862611 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:05.862582    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerStarted","Data":"557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a"}
Jul 28 18:05:05.862771 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:05.862637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerStarted","Data":"6d9c0a8a5f9d6c68de1295e210482551c9508d9af9551e57c978f18eb891e467"}
Jul 28 18:05:08.876210 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:08.876183    2571 generic.go:356] "Generic (PLEG): container finished" podID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerID="557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a" exitCode=0
Jul 28 18:05:08.876528 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:08.876309    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerDied","Data":"557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a"}
Jul 28 18:05:08.877520 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:08.877504    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:05:30.982123 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:30.982087    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerStarted","Data":"4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6"}
Jul 28 18:05:30.982637 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:30.982158    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerStarted","Data":"9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31"}
Jul 28 18:05:30.982637 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:30.982399    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:31.006737 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:31.006693    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podStartSLOduration=2.920283562 podStartE2EDuration="28.006681344s" totalImagesPullingTime="21.232932442s" totalInitContainerRuntime="3.85346534s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:31.004938419 +0000 UTC m=+3315.764964072" watchObservedRunningTime="2026-07-28 18:05:31.006681344 +0000 UTC m=+3315.766706972"
Jul 28 18:05:31.985471 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:31.985432    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:31.986335 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:31.986311    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:05:32.988798 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:32.988755    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:05:37.993597 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:37.993567    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:05:37.994085 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:37.994058    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:05:47.994365 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:47.994330    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:05:57.994736 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:05:57.994693    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:06:07.994831 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:07.994787    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:06:17.994643 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:17.994605    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:06:27.994647 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:27.994611    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 28 18:06:37.994784 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:37.994757    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:06:44.075187 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:44.075153    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"]
Jul 28 18:06:44.075634 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:44.075393    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container" containerID="cri-o://9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31" gracePeriod=30
Jul 28 18:06:44.075634 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:44.075436    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kube-rbac-proxy" containerID="cri-o://4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6" gracePeriod=30
Jul 28 18:06:44.223985 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:44.223953    2571 generic.go:356] "Generic (PLEG): container finished" podID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerID="4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6" exitCode=2
Jul 28 18:06:44.224085 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:44.224024    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerDied","Data":"4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6"}
Jul 28 18:06:47.814698 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.814676    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:06:47.857544 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857518    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0fc5db0-4cfd-4830-937b-98e01be2031d-kserve-provision-location\") pod \"f0fc5db0-4cfd-4830-937b-98e01be2031d\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") "
Jul 28 18:06:47.857656 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857559    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-cabundle-cert\") pod \"f0fc5db0-4cfd-4830-937b-98e01be2031d\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") "
Jul 28 18:06:47.857656 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857587    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pmrk\" (UniqueName: \"kubernetes.io/projected/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-api-access-8pmrk\") pod \"f0fc5db0-4cfd-4830-937b-98e01be2031d\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") "
Jul 28 18:06:47.857656 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857613    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls\") pod \"f0fc5db0-4cfd-4830-937b-98e01be2031d\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") "
Jul 28 18:06:47.857824 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857701    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-rbac-proxy-sar-config\") pod \"f0fc5db0-4cfd-4830-937b-98e01be2031d\" (UID: \"f0fc5db0-4cfd-4830-937b-98e01be2031d\") "
Jul 28 18:06:47.857873 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857830    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fc5db0-4cfd-4830-937b-98e01be2031d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0fc5db0-4cfd-4830-937b-98e01be2031d" (UID: "f0fc5db0-4cfd-4830-937b-98e01be2031d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:06:47.857993 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857912    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0fc5db0-4cfd-4830-937b-98e01be2031d-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:06:47.858059 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.857998    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0fc5db0-4cfd-4830-937b-98e01be2031d" (UID: "f0fc5db0-4cfd-4830-937b-98e01be2031d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:06:47.858059 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.858049    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0fc5db0-4cfd-4830-937b-98e01be2031d" (UID: "f0fc5db0-4cfd-4830-937b-98e01be2031d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:06:47.859510 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.859486    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-api-access-8pmrk" (OuterVolumeSpecName: "kube-api-access-8pmrk") pod "f0fc5db0-4cfd-4830-937b-98e01be2031d" (UID: "f0fc5db0-4cfd-4830-937b-98e01be2031d"). InnerVolumeSpecName "kube-api-access-8pmrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:06:47.859614 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.859597    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0fc5db0-4cfd-4830-937b-98e01be2031d" (UID: "f0fc5db0-4cfd-4830-937b-98e01be2031d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:06:47.959074 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.959024    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:06:47.959074 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.959046    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0fc5db0-4cfd-4830-937b-98e01be2031d-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:06:47.959074 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.959055    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmrk\" (UniqueName: \"kubernetes.io/projected/f0fc5db0-4cfd-4830-937b-98e01be2031d-kube-api-access-8pmrk\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:06:47.959074 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:47.959064    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0fc5db0-4cfd-4830-937b-98e01be2031d-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:06:48.239893 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.239839    2571 generic.go:356] "Generic (PLEG): container finished" podID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerID="9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31" exitCode=0
Jul 28 18:06:48.239893 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.239870    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerDied","Data":"9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31"}
Jul 28 18:06:48.240019 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.239893    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"
Jul 28 18:06:48.240019 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.239914    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs" event={"ID":"f0fc5db0-4cfd-4830-937b-98e01be2031d","Type":"ContainerDied","Data":"6d9c0a8a5f9d6c68de1295e210482551c9508d9af9551e57c978f18eb891e467"}
Jul 28 18:06:48.240019 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.239940    2571 scope.go:117] "RemoveContainer" containerID="4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6"
Jul 28 18:06:48.247603 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.247586    2571 scope.go:117] "RemoveContainer" containerID="9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31"
Jul 28 18:06:48.254517 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.254499    2571 scope.go:117] "RemoveContainer" containerID="557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a"
Jul 28 18:06:48.261445 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.261363    2571 scope.go:117] "RemoveContainer" containerID="4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6"
Jul 28 18:06:48.261700 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:06:48.261679    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6\": container with ID starting with 4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6 not found: ID does not exist" containerID="4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6"
Jul 28 18:06:48.261793 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.261709    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6"} err="failed to get container status \"4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6\": rpc error: code = NotFound desc = could not find container \"4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6\": container with ID starting with 4e4a491a45e90dd1303c2aeaf060cd15161cf8cd7834e1a20bd86091671e47e6 not found: ID does not exist"
Jul 28 18:06:48.261793 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.261728    2571 scope.go:117] "RemoveContainer" containerID="9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31"
Jul 28 18:06:48.261981 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:06:48.261963    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31\": container with ID starting with 9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31 not found: ID does not exist" containerID="9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31"
Jul 28 18:06:48.262026 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.261986    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31"} err="failed to get container status \"9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31\": rpc error: code = NotFound desc = could not find container \"9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31\": container with ID starting with 9cc9d65212f30c93a2864d04d17355d323f4cffe01adf82e0f8ffb55efe82e31 not found: ID does not exist"
Jul 28 18:06:48.262026 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.261999    2571 scope.go:117] "RemoveContainer" containerID="557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a"
Jul 28 18:06:48.262104 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.262039    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"]
Jul 28 18:06:48.262234 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:06:48.262215    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a\": container with ID starting with 557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a not found: ID does not exist" containerID="557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a"
Jul 28 18:06:48.262316 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.262262    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a"} err="failed to get container status \"557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a\": rpc error: code = NotFound desc = could not find container \"557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a\": container with ID starting with 557c9339199302a88458df35214421d49a25a93e51c12525e173b6bbdd9d962a not found: ID does not exist"
Jul 28 18:06:48.267357 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:48.267334    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hl6gs"]
Jul 28 18:06:49.248810 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.248780    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"]
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249028    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249039    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249051    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="storage-initializer"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249057    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="storage-initializer"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249070    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kube-rbac-proxy"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249075    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kube-rbac-proxy"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249112    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kube-rbac-proxy"
Jul 28 18:06:49.249154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.249119    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" containerName="kserve-container"
Jul 28 18:06:49.251991 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.251976    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.254761 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.254741    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:06:49.254978 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.254963    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:06:49.255063 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.255008    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:06:49.255917 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.255902    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:06:49.256057 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.255904    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 18:06:49.256057 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.255906    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 18:06:49.256146 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.256132    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:06:49.262056 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.262033    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"]
Jul 28 18:06:49.368404 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.368380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.368500 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.368425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjt7n\" (UniqueName: \"kubernetes.io/projected/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-api-access-cjt7n\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.368500 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.368448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.368593 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.368493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.368593 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.368527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01d579c-4210-46b5-96eb-4412dd10c4e7-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.469314 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.469286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjt7n\" (UniqueName: \"kubernetes.io/projected/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-api-access-cjt7n\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.469415 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.469327    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.469415 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.469355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.469415 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.469381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01d579c-4210-46b5-96eb-4412dd10c4e7-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.469577 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.469418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.469577 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:06:49.469545    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 28 18:06:49.469683 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:06:49.469612    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls podName:e01d579c-4210-46b5-96eb-4412dd10c4e7 nodeName:}" failed. No retries permitted until 2026-07-28 18:06:49.969591409 +0000 UTC m=+3394.729617017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-5cpm9" (UID: "e01d579c-4210-46b5-96eb-4412dd10c4e7") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 28 18:06:49.469805 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.469787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01d579c-4210-46b5-96eb-4412dd10c4e7-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.470091 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.470072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.470131 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.470072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.478969 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.478943    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjt7n\" (UniqueName: \"kubernetes.io/projected/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-api-access-cjt7n\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.706532 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.706509    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0fc5db0-4cfd-4830-937b-98e01be2031d" path="/var/lib/kubelet/pods/f0fc5db0-4cfd-4830-937b-98e01be2031d/volumes"
Jul 28 18:06:49.971925 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.971875    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:49.974015 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:49.974000    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-5cpm9\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:50.161011 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:50.160988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:50.284547 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:50.284522    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"]
Jul 28 18:06:50.286704 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:06:50.286682    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01d579c_4210_46b5_96eb_4412dd10c4e7.slice/crio-66b700519da576e54ccfaff9c17185a20863b6060ce3e9396bde87993d9c1191 WatchSource:0}: Error finding container 66b700519da576e54ccfaff9c17185a20863b6060ce3e9396bde87993d9c1191: Status 404 returned error can't find the container with id 66b700519da576e54ccfaff9c17185a20863b6060ce3e9396bde87993d9c1191
Jul 28 18:06:51.252258 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:51.252206    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerStarted","Data":"fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560"}
Jul 28 18:06:51.252430 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:51.252292    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerStarted","Data":"66b700519da576e54ccfaff9c17185a20863b6060ce3e9396bde87993d9c1191"}
Jul 28 18:06:54.262440 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:54.262368    2571 generic.go:356] "Generic (PLEG): container finished" podID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerID="fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560" exitCode=0
Jul 28 18:06:54.262440 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:54.262424    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerDied","Data":"fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560"}
Jul 28 18:06:55.267900 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:55.267867    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerStarted","Data":"e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136"}
Jul 28 18:06:55.268258 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:55.267952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerStarted","Data":"b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e"}
Jul 28 18:06:55.268258 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:55.268138    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:55.290185 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:55.290141    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podStartSLOduration=2.692189 podStartE2EDuration="6.290130346s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.597941346s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:55.289622394 +0000 UTC m=+3400.049648027" watchObservedRunningTime="2026-07-28 18:06:55.290130346 +0000 UTC m=+3400.050155977"
Jul 28 18:06:56.271728 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:56.271697    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:06:56.272709 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:56.272682    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:06:57.275317 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:06:57.275283    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:07:02.280761 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:02.280732    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:07:02.281390 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:02.281357    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:07:12.281684 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:12.281642    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:07:22.282284 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:22.282233    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:07:32.281718 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:32.281679    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:07:42.281942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:42.281902    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:07:52.282153 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:07:52.282110    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:08:02.282301 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:02.282275    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:08:09.353758 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:09.353715    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"]
Jul 28 18:08:09.354166 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:09.354008    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" containerID="cri-o://b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e" gracePeriod=30
Jul 28 18:08:09.354166 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:09.354079    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kube-rbac-proxy" containerID="cri-o://e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136" gracePeriod=30
Jul 28 18:08:09.506777 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:09.506749    2571 generic.go:356] "Generic (PLEG): container finished" podID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerID="e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136" exitCode=2
Jul 28 18:08:09.506894 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:09.506823    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerDied","Data":"e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136"}
Jul 28 18:08:12.275578 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.275540    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 28 18:08:12.281221 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.281192    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 28 18:08:12.796058 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.796037    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:08:12.812928 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.812908    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-rbac-proxy-sar-config\") pod \"e01d579c-4210-46b5-96eb-4412dd10c4e7\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") "
Jul 28 18:08:12.813027 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.812940    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-cabundle-cert\") pod \"e01d579c-4210-46b5-96eb-4412dd10c4e7\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") "
Jul 28 18:08:12.813027 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.812962    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjt7n\" (UniqueName: \"kubernetes.io/projected/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-api-access-cjt7n\") pod \"e01d579c-4210-46b5-96eb-4412dd10c4e7\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") "
Jul 28 18:08:12.813027 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.812981    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls\") pod \"e01d579c-4210-46b5-96eb-4412dd10c4e7\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") "
Jul 28 18:08:12.813151 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.813072    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01d579c-4210-46b5-96eb-4412dd10c4e7-kserve-provision-location\") pod \"e01d579c-4210-46b5-96eb-4412dd10c4e7\" (UID: \"e01d579c-4210-46b5-96eb-4412dd10c4e7\") "
Jul 28 18:08:12.813413 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.813379    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01d579c-4210-46b5-96eb-4412dd10c4e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e01d579c-4210-46b5-96eb-4412dd10c4e7" (UID: "e01d579c-4210-46b5-96eb-4412dd10c4e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:12.813413 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.813380    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e01d579c-4210-46b5-96eb-4412dd10c4e7" (UID: "e01d579c-4210-46b5-96eb-4412dd10c4e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:08:12.813551 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.813389    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e01d579c-4210-46b5-96eb-4412dd10c4e7" (UID: "e01d579c-4210-46b5-96eb-4412dd10c4e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:08:12.814969 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.814946    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e01d579c-4210-46b5-96eb-4412dd10c4e7" (UID: "e01d579c-4210-46b5-96eb-4412dd10c4e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:08:12.815035 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.814979    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-api-access-cjt7n" (OuterVolumeSpecName: "kube-api-access-cjt7n") pod "e01d579c-4210-46b5-96eb-4412dd10c4e7" (UID: "e01d579c-4210-46b5-96eb-4412dd10c4e7"). InnerVolumeSpecName "kube-api-access-cjt7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:08:12.914099 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.914049    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:08:12.914099 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.914068    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e01d579c-4210-46b5-96eb-4412dd10c4e7-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:08:12.914099 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.914077    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjt7n\" (UniqueName: \"kubernetes.io/projected/e01d579c-4210-46b5-96eb-4412dd10c4e7-kube-api-access-cjt7n\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:08:12.914099 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.914086    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e01d579c-4210-46b5-96eb-4412dd10c4e7-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:08:12.914099 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:12.914095    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e01d579c-4210-46b5-96eb-4412dd10c4e7-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:08:13.521538 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.521509    2571 generic.go:356] "Generic (PLEG): container finished" podID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerID="b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e" exitCode=0
Jul 28 18:08:13.522010 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.521561    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"
Jul 28 18:08:13.522010 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.521594    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerDied","Data":"b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e"}
Jul 28 18:08:13.522010 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.521652    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9" event={"ID":"e01d579c-4210-46b5-96eb-4412dd10c4e7","Type":"ContainerDied","Data":"66b700519da576e54ccfaff9c17185a20863b6060ce3e9396bde87993d9c1191"}
Jul 28 18:08:13.522010 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.521684    2571 scope.go:117] "RemoveContainer" containerID="e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136"
Jul 28 18:08:13.529521 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.529506    2571 scope.go:117] "RemoveContainer" containerID="b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e"
Jul 28 18:08:13.536160 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.536146    2571 scope.go:117] "RemoveContainer" containerID="fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560"
Jul 28 18:08:13.542985 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.542970    2571 scope.go:117] "RemoveContainer" containerID="e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136"
Jul 28 18:08:13.543338 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:08:13.543236    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136\": container with ID starting with e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136 not found: ID does not exist" containerID="e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136"
Jul 28 18:08:13.543436 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.543352    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136"} err="failed to get container status \"e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136\": rpc error: code = NotFound desc = could not find container \"e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136\": container with ID starting with e5097e7c648faf557b0b6e97220b0c03b690a5b2d8b91c5b14c1ad4c197b8136 not found: ID does not exist"
Jul 28 18:08:13.543436 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.543374    2571 scope.go:117] "RemoveContainer" containerID="b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e"
Jul 28 18:08:13.543605 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:08:13.543589    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e\": container with ID starting with b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e not found: ID does not exist" containerID="b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e"
Jul 28 18:08:13.543650 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.543610    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e"} err="failed to get container status \"b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e\": rpc error: code = NotFound desc = could not find container \"b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e\": container with ID starting with b1b4c65a5482cb330b64f69d1595a08947da2aaae721adf92aa00c001d73092e not found: ID does not exist"
Jul 28 18:08:13.543650 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.543624    2571 scope.go:117] "RemoveContainer" containerID="fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560"
Jul 28 18:08:13.543769 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.543748    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"]
Jul 28 18:08:13.543815 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:08:13.543787    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560\": container with ID starting with fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560 not found: ID does not exist" containerID="fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560"
Jul 28 18:08:13.543815 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.543800    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560"} err="failed to get container status \"fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560\": rpc error: code = NotFound desc = could not find container \"fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560\": container with ID starting with fe8a5cf0a0573cce62e9b3a851ec01afca02b6d382c1e88dde776eb229022560 not found: ID does not exist"
Jul 28 18:08:13.547271 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.547233    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-5cpm9"]
Jul 28 18:08:13.706957 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:08:13.706932    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" path="/var/lib/kubelet/pods/e01d579c-4210-46b5-96eb-4412dd10c4e7/volumes"
Jul 28 18:11:46.117516 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117482    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"]
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117710    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="storage-initializer"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117720    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="storage-initializer"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117734    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117739    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117745    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kube-rbac-proxy"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117751    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kube-rbac-proxy"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117791    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kube-rbac-proxy"
Jul 28 18:11:46.117908 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.117801    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01d579c-4210-46b5-96eb-4412dd10c4e7" containerName="kserve-container"
Jul 28 18:11:46.119739 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.119725    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.122328 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.122306    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:11:46.122466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.122312    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:11:46.122466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.122407    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:11:46.122466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.122431    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:11:46.122610 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.122464    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:11:46.123393 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.123377    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 18:11:46.123475 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.123377    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:11:46.131919 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.131900    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"]
Jul 28 18:11:46.295404 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.295379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.295511 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.295408    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wnd\" (UniqueName: \"kubernetes.io/projected/31562205-0869-42f2-8746-db86458f0ea4-kube-api-access-q7wnd\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.295511 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.295427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31562205-0869-42f2-8746-db86458f0ea4-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.295511 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.295490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.295656 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.295528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31562205-0869-42f2-8746-db86458f0ea4-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396369 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396463 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wnd\" (UniqueName: \"kubernetes.io/projected/31562205-0869-42f2-8746-db86458f0ea4-kube-api-access-q7wnd\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396463 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31562205-0869-42f2-8746-db86458f0ea4-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396463 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396421    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396463 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31562205-0869-42f2-8746-db86458f0ea4-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396859 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396839    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31562205-0869-42f2-8746-db86458f0ea4-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.396975 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.396961    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.397107 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.397091    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.398854 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.398838    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31562205-0869-42f2-8746-db86458f0ea4-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.405724 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.405705    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wnd\" (UniqueName: \"kubernetes.io/projected/31562205-0869-42f2-8746-db86458f0ea4-kube-api-access-q7wnd\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.429497 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.429480    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:46.554996 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:46.554919    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"]
Jul 28 18:11:46.558593 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:11:46.558567    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31562205_0869_42f2_8746_db86458f0ea4.slice/crio-32eef04d3ccf774506a269487e0f90e131d79a4c1f76139a68b8c0e3c75b85e3 WatchSource:0}: Error finding container 32eef04d3ccf774506a269487e0f90e131d79a4c1f76139a68b8c0e3c75b85e3: Status 404 returned error can't find the container with id 32eef04d3ccf774506a269487e0f90e131d79a4c1f76139a68b8c0e3c75b85e3
Jul 28 18:11:47.178137 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:47.178100    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerStarted","Data":"e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3"}
Jul 28 18:11:47.178618 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:47.178177    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerStarted","Data":"32eef04d3ccf774506a269487e0f90e131d79a4c1f76139a68b8c0e3c75b85e3"}
Jul 28 18:11:51.197445 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:51.197415    2571 generic.go:356] "Generic (PLEG): container finished" podID="31562205-0869-42f2-8746-db86458f0ea4" containerID="e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3" exitCode=0
Jul 28 18:11:51.197789 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:51.197470    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerDied","Data":"e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3"}
Jul 28 18:11:52.202741 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:52.202709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerStarted","Data":"a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91"}
Jul 28 18:11:52.203121 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:52.202768    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerStarted","Data":"599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9"}
Jul 28 18:11:52.203121 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:52.202950    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:52.225409 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:52.225356    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podStartSLOduration=2.5764693039999997 podStartE2EDuration="6.225342616s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648873312s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:52.224856328 +0000 UTC m=+3696.984881958" watchObservedRunningTime="2026-07-28 18:11:52.225342616 +0000 UTC m=+3696.985368245"
Jul 28 18:11:53.205562 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:53.205520    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:11:59.215009 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:11:59.214975    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:12:29.215757 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:12:29.215723    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 18:12:39.216586 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:12:39.216545    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 18:12:49.216181 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:12:49.216143    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 18:12:59.251950 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:12:59.251915    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:13:06.198931 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:06.198900    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"]
Jul 28 18:13:06.199318 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:06.199117    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container" containerID="cri-o://599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9" gracePeriod=30
Jul 28 18:13:06.199318 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:06.199177    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kube-rbac-proxy" containerID="cri-o://a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91" gracePeriod=30
Jul 28 18:13:06.443388 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:06.443359    2571 generic.go:356] "Generic (PLEG): container finished" podID="31562205-0869-42f2-8746-db86458f0ea4" containerID="a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91" exitCode=2
Jul 28 18:13:06.443523 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:06.443433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerDied","Data":"a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91"}
Jul 28 18:13:09.209453 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.209371    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 28 18:13:09.215804 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.215771    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 28 18:13:09.838329 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.838306    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:13:09.955788 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.955732    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-kube-rbac-proxy-sar-config\") pod \"31562205-0869-42f2-8746-db86458f0ea4\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") "
Jul 28 18:13:09.955788 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.955760    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-cabundle-cert\") pod \"31562205-0869-42f2-8746-db86458f0ea4\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") "
Jul 28 18:13:09.955933 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.955803    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7wnd\" (UniqueName: \"kubernetes.io/projected/31562205-0869-42f2-8746-db86458f0ea4-kube-api-access-q7wnd\") pod \"31562205-0869-42f2-8746-db86458f0ea4\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") "
Jul 28 18:13:09.955933 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.955824    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31562205-0869-42f2-8746-db86458f0ea4-kserve-provision-location\") pod \"31562205-0869-42f2-8746-db86458f0ea4\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") "
Jul 28 18:13:09.955933 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.955853    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31562205-0869-42f2-8746-db86458f0ea4-proxy-tls\") pod \"31562205-0869-42f2-8746-db86458f0ea4\" (UID: \"31562205-0869-42f2-8746-db86458f0ea4\") "
Jul 28 18:13:09.956152 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.956125    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31562205-0869-42f2-8746-db86458f0ea4" (UID: "31562205-0869-42f2-8746-db86458f0ea4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:09.956221 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.956143    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31562205-0869-42f2-8746-db86458f0ea4" (UID: "31562205-0869-42f2-8746-db86458f0ea4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:09.956221 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.956144    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31562205-0869-42f2-8746-db86458f0ea4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31562205-0869-42f2-8746-db86458f0ea4" (UID: "31562205-0869-42f2-8746-db86458f0ea4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:09.957833 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.957808    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31562205-0869-42f2-8746-db86458f0ea4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31562205-0869-42f2-8746-db86458f0ea4" (UID: "31562205-0869-42f2-8746-db86458f0ea4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:13:09.957929 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:09.957858    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31562205-0869-42f2-8746-db86458f0ea4-kube-api-access-q7wnd" (OuterVolumeSpecName: "kube-api-access-q7wnd") pod "31562205-0869-42f2-8746-db86458f0ea4" (UID: "31562205-0869-42f2-8746-db86458f0ea4"). InnerVolumeSpecName "kube-api-access-q7wnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:13:10.057000 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.056976    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7wnd\" (UniqueName: \"kubernetes.io/projected/31562205-0869-42f2-8746-db86458f0ea4-kube-api-access-q7wnd\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:13:10.057000 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.056997    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31562205-0869-42f2-8746-db86458f0ea4-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:13:10.057133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.057007    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31562205-0869-42f2-8746-db86458f0ea4-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:13:10.057133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.057017    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:13:10.057133 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.057027    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31562205-0869-42f2-8746-db86458f0ea4-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:13:10.458924 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.458896    2571 generic.go:356] "Generic (PLEG): container finished" podID="31562205-0869-42f2-8746-db86458f0ea4" containerID="599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9" exitCode=0
Jul 28 18:13:10.459225 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.458943    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerDied","Data":"599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9"}
Jul 28 18:13:10.459225 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.459002    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj" event={"ID":"31562205-0869-42f2-8746-db86458f0ea4","Type":"ContainerDied","Data":"32eef04d3ccf774506a269487e0f90e131d79a4c1f76139a68b8c0e3c75b85e3"}
Jul 28 18:13:10.459225 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.458953    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"
Jul 28 18:13:10.459225 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.459031    2571 scope.go:117] "RemoveContainer" containerID="a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91"
Jul 28 18:13:10.466903 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.466722    2571 scope.go:117] "RemoveContainer" containerID="599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9"
Jul 28 18:13:10.473485 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.473470    2571 scope.go:117] "RemoveContainer" containerID="e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3"
Jul 28 18:13:10.479732 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.479719    2571 scope.go:117] "RemoveContainer" containerID="a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91"
Jul 28 18:13:10.479964 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:13:10.479948    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91\": container with ID starting with a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91 not found: ID does not exist" containerID="a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91"
Jul 28 18:13:10.480013 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.479972    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91"} err="failed to get container status \"a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91\": rpc error: code = NotFound desc = could not find container \"a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91\": container with ID starting with a98aa1b3f3d85b93879cfd8d67efc61c4498cd084997bf2471cdb386a4219a91 not found: ID does not exist"
Jul 28 18:13:10.480013 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.479988    2571 scope.go:117] "RemoveContainer" containerID="599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9"
Jul 28 18:13:10.480194 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:13:10.480179    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9\": container with ID starting with 599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9 not found: ID does not exist" containerID="599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9"
Jul 28 18:13:10.480234 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.480198    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9"} err="failed to get container status \"599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9\": rpc error: code = NotFound desc = could not find container \"599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9\": container with ID starting with 599e7d42f18403bc6a4d1212c56f3b0c4677d6b1cdb0958ef7ed858f86200ed9 not found: ID does not exist"
Jul 28 18:13:10.480234 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.480211    2571 scope.go:117] "RemoveContainer" containerID="e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3"
Jul 28 18:13:10.480419 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:13:10.480401    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3\": container with ID starting with e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3 not found: ID does not exist" containerID="e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3"
Jul 28 18:13:10.480477 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.480426    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3"} err="failed to get container status \"e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3\": rpc error: code = NotFound desc = could not find container \"e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3\": container with ID starting with e8a6ac343aa3e92328c274c15fbe644727650789a92ad319220bb76c51a46ad3 not found: ID does not exist"
Jul 28 18:13:10.482614 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.482592    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"]
Jul 28 18:13:10.486652 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:10.486633    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-hzvbj"]
Jul 28 18:13:11.710955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:13:11.709097    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31562205-0869-42f2-8746-db86458f0ea4" path="/var/lib/kubelet/pods/31562205-0869-42f2-8746-db86458f0ea4/volumes"
Jul 28 18:19:07.009557 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009525    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"]
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009748    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009759    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009773    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kube-rbac-proxy"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009778    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kube-rbac-proxy"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009786    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="storage-initializer"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009792    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="storage-initializer"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009829    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kserve-container"
Jul 28 18:19:07.009955 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.009837    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="31562205-0869-42f2-8746-db86458f0ea4" containerName="kube-rbac-proxy"
Jul 28 18:19:07.012563 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.012548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.015009 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.014984    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 18:19:07.015225 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.015205    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:19:07.015365 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.015345    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:19:07.015448 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.015360    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:19:07.015448 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.015426    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:19:07.015448 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.015354    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:19:07.015448 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.015361    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:19:07.022724 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.022705    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"]
Jul 28 18:19:07.117426 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.117403    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7jt\" (UniqueName: \"kubernetes.io/projected/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-api-access-ch7jt\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.117527 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.117429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a638fe-a14b-402a-b088-b67a3e83bd96-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.117527 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.117449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a638fe-a14b-402a-b088-b67a3e83bd96-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.117527 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.117475    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.117527 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.117522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.217833 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.217811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.217930 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.217840    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.217930 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.217866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7jt\" (UniqueName: \"kubernetes.io/projected/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-api-access-ch7jt\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.217930 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.217883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a638fe-a14b-402a-b088-b67a3e83bd96-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.217930 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.217901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a638fe-a14b-402a-b088-b67a3e83bd96-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.218321 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.218305    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a638fe-a14b-402a-b088-b67a3e83bd96-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.218533 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.218516    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.218533 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.218526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.220309 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.220290    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a638fe-a14b-402a-b088-b67a3e83bd96-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.226187 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.226166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7jt\" (UniqueName: \"kubernetes.io/projected/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-api-access-ch7jt\") pod \"isvc-sklearn-v2-predictor-c6d677456-g9ltl\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.321766 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.321707    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:07.442956 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.442934    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"]
Jul 28 18:19:07.444976 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:19:07.444951    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a638fe_a14b_402a_b088_b67a3e83bd96.slice/crio-cd375ea81531c5637bc6dc3ed73220124ce65cbba791f322b5c3bc0204303002 WatchSource:0}: Error finding container cd375ea81531c5637bc6dc3ed73220124ce65cbba791f322b5c3bc0204303002: Status 404 returned error can't find the container with id cd375ea81531c5637bc6dc3ed73220124ce65cbba791f322b5c3bc0204303002
Jul 28 18:19:07.561044 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.561015    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerStarted","Data":"5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf"}
Jul 28 18:19:07.561143 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:07.561075    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerStarted","Data":"cd375ea81531c5637bc6dc3ed73220124ce65cbba791f322b5c3bc0204303002"}
Jul 28 18:19:11.578139 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:11.578107    2571 generic.go:356] "Generic (PLEG): container finished" podID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerID="5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf" exitCode=0
Jul 28 18:19:11.578456 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:11.578156    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerDied","Data":"5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf"}
Jul 28 18:19:12.583432 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:12.583399    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerStarted","Data":"dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c"}
Jul 28 18:19:12.583877 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:12.583467    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerStarted","Data":"348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc"}
Jul 28 18:19:12.583877 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:12.583689    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:12.605633 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:12.605590    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podStartSLOduration=3.037584474 podStartE2EDuration="6.605578527s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.567994053s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:12.604119097 +0000 UTC m=+4137.364144725" watchObservedRunningTime="2026-07-28 18:19:12.605578527 +0000 UTC m=+4137.365604154"
Jul 28 18:19:13.587144 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:13.587113    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:13.588152 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:13.588125    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:19:14.590217 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:14.590176    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:19:19.594885 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:19.594858    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:19:19.595317 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:19.595293    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:19:29.595435 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:29.595398    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:19:39.596122 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:39.596083    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:19:49.595326 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:49.595289    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:19:59.595289 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:19:59.595232    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:20:09.595593 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:09.595554    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:20:19.596386 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:19.596355    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:20:27.214396 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:27.214367    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"]
Jul 28 18:20:27.214950 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:27.214595    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" containerID="cri-o://348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc" gracePeriod=30
Jul 28 18:20:27.214950 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:27.214673    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kube-rbac-proxy" containerID="cri-o://dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c" gracePeriod=30
Jul 28 18:20:27.823758 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:27.823732    2571 generic.go:356] "Generic (PLEG): container finished" podID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerID="dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c" exitCode=2
Jul 28 18:20:27.823904 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:27.823802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerDied","Data":"dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c"}
Jul 28 18:20:29.590500 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:29.590467    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Jul 28 18:20:29.596193 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:29.596166    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 28 18:20:30.754028 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.754006    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:20:30.835468 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.835403    2571 generic.go:356] "Generic (PLEG): container finished" podID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerID="348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc" exitCode=0
Jul 28 18:20:30.835468 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.835460    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"
Jul 28 18:20:30.835646 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.835495    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerDied","Data":"348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc"}
Jul 28 18:20:30.835646 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.835565    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl" event={"ID":"34a638fe-a14b-402a-b088-b67a3e83bd96","Type":"ContainerDied","Data":"cd375ea81531c5637bc6dc3ed73220124ce65cbba791f322b5c3bc0204303002"}
Jul 28 18:20:30.835646 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.835606    2571 scope.go:117] "RemoveContainer" containerID="dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c"
Jul 28 18:20:30.842788 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.842770    2571 scope.go:117] "RemoveContainer" containerID="348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc"
Jul 28 18:20:30.849373 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.849357    2571 scope.go:117] "RemoveContainer" containerID="5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf"
Jul 28 18:20:30.849888 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.849869    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-rbac-proxy-sar-config\") pod \"34a638fe-a14b-402a-b088-b67a3e83bd96\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") "
Jul 28 18:20:30.849977 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.849909    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a638fe-a14b-402a-b088-b67a3e83bd96-kserve-provision-location\") pod \"34a638fe-a14b-402a-b088-b67a3e83bd96\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") "
Jul 28 18:20:30.849977 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.849932    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-cabundle-cert\") pod \"34a638fe-a14b-402a-b088-b67a3e83bd96\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") "
Jul 28 18:20:30.850077 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.849987    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch7jt\" (UniqueName: \"kubernetes.io/projected/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-api-access-ch7jt\") pod \"34a638fe-a14b-402a-b088-b67a3e83bd96\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") "
Jul 28 18:20:30.850077 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.850013    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a638fe-a14b-402a-b088-b67a3e83bd96-proxy-tls\") pod \"34a638fe-a14b-402a-b088-b67a3e83bd96\" (UID: \"34a638fe-a14b-402a-b088-b67a3e83bd96\") "
Jul 28 18:20:30.850319 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.850229    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a638fe-a14b-402a-b088-b67a3e83bd96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34a638fe-a14b-402a-b088-b67a3e83bd96" (UID: "34a638fe-a14b-402a-b088-b67a3e83bd96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:30.850319 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.850274    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34a638fe-a14b-402a-b088-b67a3e83bd96" (UID: "34a638fe-a14b-402a-b088-b67a3e83bd96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:20:30.850415 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.850401    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:20:30.850472 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.850418    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a638fe-a14b-402a-b088-b67a3e83bd96-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:20:30.850677 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.850658    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34a638fe-a14b-402a-b088-b67a3e83bd96" (UID: "34a638fe-a14b-402a-b088-b67a3e83bd96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:20:30.851965 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.851942    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a638fe-a14b-402a-b088-b67a3e83bd96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34a638fe-a14b-402a-b088-b67a3e83bd96" (UID: "34a638fe-a14b-402a-b088-b67a3e83bd96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:20:30.852031 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.851978    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-api-access-ch7jt" (OuterVolumeSpecName: "kube-api-access-ch7jt") pod "34a638fe-a14b-402a-b088-b67a3e83bd96" (UID: "34a638fe-a14b-402a-b088-b67a3e83bd96"). InnerVolumeSpecName "kube-api-access-ch7jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:20:30.862638 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.862612    2571 scope.go:117] "RemoveContainer" containerID="dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c"
Jul 28 18:20:30.862875 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:20:30.862859    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c\": container with ID starting with dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c not found: ID does not exist" containerID="dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c"
Jul 28 18:20:30.862934 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.862882    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c"} err="failed to get container status \"dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c\": rpc error: code = NotFound desc = could not find container \"dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c\": container with ID starting with dd9c21e9ce283a1ed2c3a088c4696628f52c07ffd4a729c7eaaaa52b9c3e4d5c not found: ID does not exist"
Jul 28 18:20:30.862934 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.862899    2571 scope.go:117] "RemoveContainer" containerID="348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc"
Jul 28 18:20:30.863129 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:20:30.863114    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc\": container with ID starting with 348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc not found: ID does not exist" containerID="348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc"
Jul 28 18:20:30.863186 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.863132    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc"} err="failed to get container status \"348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc\": rpc error: code = NotFound desc = could not find container \"348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc\": container with ID starting with 348c829d8df88dacf4795bcf241aa389ea9b3ce54a14f85cfd2146e3a624d6cc not found: ID does not exist"
Jul 28 18:20:30.863186 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.863144    2571 scope.go:117] "RemoveContainer" containerID="5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf"
Jul 28 18:20:30.863414 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:20:30.863396    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf\": container with ID starting with 5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf not found: ID does not exist" containerID="5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf"
Jul 28 18:20:30.863460 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.863423    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf"} err="failed to get container status \"5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf\": rpc error: code = NotFound desc = could not find container \"5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf\": container with ID starting with 5a60b5b6e3197b29996d3c04b0797338e70a8ef7a435d5b7cba3c3c42bc2ecbf not found: ID does not exist"
Jul 28 18:20:30.950890 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.950869    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7jt\" (UniqueName: \"kubernetes.io/projected/34a638fe-a14b-402a-b088-b67a3e83bd96-kube-api-access-ch7jt\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:20:30.950890 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.950888    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34a638fe-a14b-402a-b088-b67a3e83bd96-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:20:30.951005 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:30.950899    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a638fe-a14b-402a-b088-b67a3e83bd96-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:20:31.159738 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:31.159714    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"]
Jul 28 18:20:31.164073 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:31.164052    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-g9ltl"]
Jul 28 18:20:31.706306 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:20:31.706278    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" path="/var/lib/kubelet/pods/34a638fe-a14b-402a-b088-b67a3e83bd96/volumes"
Jul 28 18:22:07.804036 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804004    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"]
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804225    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kube-rbac-proxy"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804236    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kube-rbac-proxy"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804258    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="storage-initializer"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804263    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="storage-initializer"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804270    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804276    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804319    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kube-rbac-proxy"
Jul 28 18:22:07.804442 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.804327    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a638fe-a14b-402a-b088-b67a3e83bd96" containerName="kserve-container"
Jul 28 18:22:07.806187 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.806171    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.808972 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.808940    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:22:07.808972 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.808953    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:22:07.808972 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.808940    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 18:22:07.809231 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.809037    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:22:07.809321 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.809270    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 18:22:07.809321 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.809276    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:22:07.810042 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.810027    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:22:07.817960 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.817938    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"]
Jul 28 18:22:07.863962 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.863938    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.864063 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.863976    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.864063 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.864007    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.864063 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.864046    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtm5\" (UniqueName: \"kubernetes.io/projected/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-api-access-2vtm5\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.864186 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.864077    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965301 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965276    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965411 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtm5\" (UniqueName: \"kubernetes.io/projected/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-api-access-2vtm5\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965411 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965338    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965411 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965568 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965771 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965899 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965881    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.965998 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.965972    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.967637 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.967617    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:07.975167 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:07.975142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtm5\" (UniqueName: \"kubernetes.io/projected/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-api-access-2vtm5\") pod \"isvc-tensorflow-predictor-547dfc5669-89pkd\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:08.116108 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:08.116057    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:08.240968 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:08.240944    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"]
Jul 28 18:22:08.243272 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:22:08.243231    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f2db8f_1613_45ea_a5d4_3a79ecdafb83.slice/crio-b91fe9770fc09986b99457715886d6ea00ff63721d30812ea92555b91292502b WatchSource:0}: Error finding container b91fe9770fc09986b99457715886d6ea00ff63721d30812ea92555b91292502b: Status 404 returned error can't find the container with id b91fe9770fc09986b99457715886d6ea00ff63721d30812ea92555b91292502b
Jul 28 18:22:09.137962 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:09.137931    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerStarted","Data":"76928d1da6fd5c9b8f71cc8fb310e219cbe1c8dcd399b9514f2843768b062346"}
Jul 28 18:22:09.138321 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:09.137984    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerStarted","Data":"b91fe9770fc09986b99457715886d6ea00ff63721d30812ea92555b91292502b"}
Jul 28 18:22:13.154546 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:13.154452    2571 generic.go:356] "Generic (PLEG): container finished" podID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerID="76928d1da6fd5c9b8f71cc8fb310e219cbe1c8dcd399b9514f2843768b062346" exitCode=0
Jul 28 18:22:13.154546 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:13.154542    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerDied","Data":"76928d1da6fd5c9b8f71cc8fb310e219cbe1c8dcd399b9514f2843768b062346"}
Jul 28 18:22:13.155713 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:13.155697    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:22:17.172777 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:17.172746    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerStarted","Data":"64abd6935dbe04dbb6bb595b7b01d746336632f3eda1c988f42521b5eec53219"}
Jul 28 18:22:17.173204 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:17.172814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerStarted","Data":"b24366f377d8044b13217f258046be3ab1174a7a7e5556978f7fe8932700a471"}
Jul 28 18:22:17.173204 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:17.173043    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:17.199174 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:17.199128    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podStartSLOduration=2.016889042 podStartE2EDuration="10.199115918s" totalImagesPullingTime="3.673243128s" totalInitContainerRuntime="4.508983748s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:22:17.196936894 +0000 UTC m=+4321.956962524" watchObservedRunningTime="2026-07-28 18:22:17.199115918 +0000 UTC m=+4321.959141545"
Jul 28 18:22:18.176160 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:18.176131    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:18.177293 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:18.177262    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 18:22:19.179401 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:19.179356    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 18:22:24.184436 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:24.184400    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:24.184924 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:24.184896    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 28 18:22:34.185192 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:34.185166    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:45.994199 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:45.994166    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"]
Jul 28 18:22:45.995670 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:45.994421    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container" containerID="cri-o://b24366f377d8044b13217f258046be3ab1174a7a7e5556978f7fe8932700a471" gracePeriod=30
Jul 28 18:22:45.995670 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:45.994459    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" containerID="cri-o://64abd6935dbe04dbb6bb595b7b01d746336632f3eda1c988f42521b5eec53219" gracePeriod=30
Jul 28 18:22:46.267795 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:46.267735    2571 generic.go:356] "Generic (PLEG): container finished" podID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerID="64abd6935dbe04dbb6bb595b7b01d746336632f3eda1c988f42521b5eec53219" exitCode=2
Jul 28 18:22:46.267881 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:46.267803    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerDied","Data":"64abd6935dbe04dbb6bb595b7b01d746336632f3eda1c988f42521b5eec53219"}
Jul 28 18:22:49.179732 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:49.179684    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 18:22:51.194476 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.194443    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"]
Jul 28 18:22:51.196748 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.196734    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.199360 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.199339    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 18:22:51.199460 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.199363    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:22:51.206784 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.206761    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"]
Jul 28 18:22:51.251754 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.251734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.251851 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.251771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.251851 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.251791    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438fa904-0ae8-4f8c-95d3-574067d4e5c5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.251851 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.251812    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z77m\" (UniqueName: \"kubernetes.io/projected/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-api-access-4z77m\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.251956 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.251858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352299 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352410 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352410 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438fa904-0ae8-4f8c-95d3-574067d4e5c5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352410 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352365    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z77m\" (UniqueName: \"kubernetes.io/projected/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-api-access-4z77m\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352410 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352796 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.352980 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.352960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.353110 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.353094    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.354753 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.354734    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438fa904-0ae8-4f8c-95d3-574067d4e5c5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.360729 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.360701    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z77m\" (UniqueName: \"kubernetes.io/projected/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-api-access-4z77m\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.506454 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.506399    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:51.630901 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:51.630873    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"]
Jul 28 18:22:51.633435 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:22:51.633411    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438fa904_0ae8_4f8c_95d3_574067d4e5c5.slice/crio-80effc5ab84e78798bade250bcb53921ad80e8aa558ed92203509b49b6033f2a WatchSource:0}: Error finding container 80effc5ab84e78798bade250bcb53921ad80e8aa558ed92203509b49b6033f2a: Status 404 returned error can't find the container with id 80effc5ab84e78798bade250bcb53921ad80e8aa558ed92203509b49b6033f2a
Jul 28 18:22:52.289145 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:52.289110    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerStarted","Data":"d13f8ac2687a852113512f2d93cef2b51e70930c0cba3bcd6b6ae6280e9005f2"}
Jul 28 18:22:52.289516 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:52.289169    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerStarted","Data":"80effc5ab84e78798bade250bcb53921ad80e8aa558ed92203509b49b6033f2a"}
Jul 28 18:22:54.180020 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:54.179983    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 18:22:56.302883 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:56.302854    2571 generic.go:356] "Generic (PLEG): container finished" podID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerID="d13f8ac2687a852113512f2d93cef2b51e70930c0cba3bcd6b6ae6280e9005f2" exitCode=0
Jul 28 18:22:56.303174 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:56.302893    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerDied","Data":"d13f8ac2687a852113512f2d93cef2b51e70930c0cba3bcd6b6ae6280e9005f2"}
Jul 28 18:22:57.308477 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:57.308443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerStarted","Data":"be0d0196fdd82c0f8156fd20d6fd57b336129b6a555309563e326055f4236d3b"}
Jul 28 18:22:57.308839 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:57.308502    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerStarted","Data":"265690aa462eb7f08d16c1e39ee0687a04c2e71e7fc2d403053242e6a4b0ad25"}
Jul 28 18:22:57.308839 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:57.308682    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:57.331085 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:57.331044    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podStartSLOduration=1.944667748 podStartE2EDuration="6.331031307s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.386363559s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:22:57.329847412 +0000 UTC m=+4362.089873040" watchObservedRunningTime="2026-07-28 18:22:57.331031307 +0000 UTC m=+4362.091056934"
Jul 28 18:22:58.311519 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:58.311492    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:22:58.312484 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:58.312458    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 18:22:59.179575 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:59.179536    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 18:22:59.179738 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:59.179654    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:22:59.314934 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:22:59.314901    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 18:23:04.180012 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:04.179974    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 18:23:04.319078 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:04.319048    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:23:04.319506 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:04.319482    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 28 18:23:09.180098 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:09.180053    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 18:23:14.180156 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:14.180113    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 28 18:23:14.320188 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:14.320166    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:23:16.372453 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.372396    2571 generic.go:356] "Generic (PLEG): container finished" podID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerID="b24366f377d8044b13217f258046be3ab1174a7a7e5556978f7fe8932700a471" exitCode=137
Jul 28 18:23:16.372453 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.372435    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerDied","Data":"b24366f377d8044b13217f258046be3ab1174a7a7e5556978f7fe8932700a471"}
Jul 28 18:23:16.631526 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.631476    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:23:16.715308 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715283    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-cabundle-cert\") pod \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") "
Jul 28 18:23:16.715424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715311    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-rbac-proxy-sar-config\") pod \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") "
Jul 28 18:23:16.715424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715359    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kserve-provision-location\") pod \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") "
Jul 28 18:23:16.715424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715385    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-proxy-tls\") pod \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") "
Jul 28 18:23:16.715424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715419    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vtm5\" (UniqueName: \"kubernetes.io/projected/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-api-access-2vtm5\") pod \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\" (UID: \"41f2db8f-1613-45ea-a5d4-3a79ecdafb83\") "
Jul 28 18:23:16.715731 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715704    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41f2db8f-1613-45ea-a5d4-3a79ecdafb83" (UID: "41f2db8f-1613-45ea-a5d4-3a79ecdafb83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:23:16.715789 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.715727    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41f2db8f-1613-45ea-a5d4-3a79ecdafb83" (UID: "41f2db8f-1613-45ea-a5d4-3a79ecdafb83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:23:16.717513 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.717489    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41f2db8f-1613-45ea-a5d4-3a79ecdafb83" (UID: "41f2db8f-1613-45ea-a5d4-3a79ecdafb83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:23:16.717850 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.717828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-api-access-2vtm5" (OuterVolumeSpecName: "kube-api-access-2vtm5") pod "41f2db8f-1613-45ea-a5d4-3a79ecdafb83" (UID: "41f2db8f-1613-45ea-a5d4-3a79ecdafb83"). InnerVolumeSpecName "kube-api-access-2vtm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:23:16.730116 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.730093    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41f2db8f-1613-45ea-a5d4-3a79ecdafb83" (UID: "41f2db8f-1613-45ea-a5d4-3a79ecdafb83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:16.816386 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.816370    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:23:16.816458 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.816387    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:23:16.816458 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.816398    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:23:16.816458 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.816407    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:23:16.816458 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:16.816415    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vtm5\" (UniqueName: \"kubernetes.io/projected/41f2db8f-1613-45ea-a5d4-3a79ecdafb83-kube-api-access-2vtm5\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:23:17.377862 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.377834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd" event={"ID":"41f2db8f-1613-45ea-a5d4-3a79ecdafb83","Type":"ContainerDied","Data":"b91fe9770fc09986b99457715886d6ea00ff63721d30812ea92555b91292502b"}
Jul 28 18:23:17.378367 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.377881    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"
Jul 28 18:23:17.378367 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.377897    2571 scope.go:117] "RemoveContainer" containerID="64abd6935dbe04dbb6bb595b7b01d746336632f3eda1c988f42521b5eec53219"
Jul 28 18:23:17.385663 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.385290    2571 scope.go:117] "RemoveContainer" containerID="b24366f377d8044b13217f258046be3ab1174a7a7e5556978f7fe8932700a471"
Jul 28 18:23:17.392437 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.392422    2571 scope.go:117] "RemoveContainer" containerID="76928d1da6fd5c9b8f71cc8fb310e219cbe1c8dcd399b9514f2843768b062346"
Jul 28 18:23:17.400235 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.400211    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"]
Jul 28 18:23:17.404449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.404428    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-89pkd"]
Jul 28 18:23:17.706353 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:17.706331    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" path="/var/lib/kubelet/pods/41f2db8f-1613-45ea-a5d4-3a79ecdafb83/volumes"
Jul 28 18:23:29.416172 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:29.416133    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"]
Jul 28 18:23:29.416601 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:29.416452    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container" containerID="cri-o://265690aa462eb7f08d16c1e39ee0687a04c2e71e7fc2d403053242e6a4b0ad25" gracePeriod=30
Jul 28 18:23:29.416601 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:29.416492    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" containerID="cri-o://be0d0196fdd82c0f8156fd20d6fd57b336129b6a555309563e326055f4236d3b" gracePeriod=30
Jul 28 18:23:30.425710 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:30.425682    2571 generic.go:356] "Generic (PLEG): container finished" podID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerID="be0d0196fdd82c0f8156fd20d6fd57b336129b6a555309563e326055f4236d3b" exitCode=2
Jul 28 18:23:30.426017 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:30.425738    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerDied","Data":"be0d0196fdd82c0f8156fd20d6fd57b336129b6a555309563e326055f4236d3b"}
Jul 28 18:23:34.315527 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:34.315486    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 28 18:23:39.315716 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:39.315633    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 28 18:23:44.316083 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:44.316045    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 28 18:23:44.316525 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:44.316156    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:23:49.315488 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:49.315452    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 28 18:23:54.315592 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:54.315558    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 28 18:23:59.315514 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:59.315478    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 28 18:23:59.517991 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:59.517965    2571 generic.go:356] "Generic (PLEG): container finished" podID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerID="265690aa462eb7f08d16c1e39ee0687a04c2e71e7fc2d403053242e6a4b0ad25" exitCode=137
Jul 28 18:23:59.518092 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:23:59.518041    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerDied","Data":"265690aa462eb7f08d16c1e39ee0687a04c2e71e7fc2d403053242e6a4b0ad25"}
Jul 28 18:24:00.047015 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.046991    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:24:00.203333 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203279    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z77m\" (UniqueName: \"kubernetes.io/projected/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-api-access-4z77m\") pod \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") "
Jul 28 18:24:00.203333 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203311    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438fa904-0ae8-4f8c-95d3-574067d4e5c5-proxy-tls\") pod \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") "
Jul 28 18:24:00.203556 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203347    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-cabundle-cert\") pod \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") "
Jul 28 18:24:00.203556 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203373    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kserve-provision-location\") pod \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") "
Jul 28 18:24:00.203556 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203418    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-rbac-proxy-sar-config\") pod \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\" (UID: \"438fa904-0ae8-4f8c-95d3-574067d4e5c5\") "
Jul 28 18:24:00.203769 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203669    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "438fa904-0ae8-4f8c-95d3-574067d4e5c5" (UID: "438fa904-0ae8-4f8c-95d3-574067d4e5c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:00.203830 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.203802    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "438fa904-0ae8-4f8c-95d3-574067d4e5c5" (UID: "438fa904-0ae8-4f8c-95d3-574067d4e5c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:00.205233 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.205209    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-api-access-4z77m" (OuterVolumeSpecName: "kube-api-access-4z77m") pod "438fa904-0ae8-4f8c-95d3-574067d4e5c5" (UID: "438fa904-0ae8-4f8c-95d3-574067d4e5c5"). InnerVolumeSpecName "kube-api-access-4z77m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:24:00.205343 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.205271    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438fa904-0ae8-4f8c-95d3-574067d4e5c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "438fa904-0ae8-4f8c-95d3-574067d4e5c5" (UID: "438fa904-0ae8-4f8c-95d3-574067d4e5c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:24:00.216999 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.216974    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "438fa904-0ae8-4f8c-95d3-574067d4e5c5" (UID: "438fa904-0ae8-4f8c-95d3-574067d4e5c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:24:00.304160 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.304138    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/438fa904-0ae8-4f8c-95d3-574067d4e5c5-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:24:00.304160 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.304159    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:24:00.304294 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.304167    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:24:00.304294 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.304177    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:24:00.304294 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.304185    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z77m\" (UniqueName: \"kubernetes.io/projected/438fa904-0ae8-4f8c-95d3-574067d4e5c5-kube-api-access-4z77m\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:24:00.525924 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.525867    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp" event={"ID":"438fa904-0ae8-4f8c-95d3-574067d4e5c5","Type":"ContainerDied","Data":"80effc5ab84e78798bade250bcb53921ad80e8aa558ed92203509b49b6033f2a"}
Jul 28 18:24:00.526280 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.525930    2571 scope.go:117] "RemoveContainer" containerID="be0d0196fdd82c0f8156fd20d6fd57b336129b6a555309563e326055f4236d3b"
Jul 28 18:24:00.526280 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.525877    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"
Jul 28 18:24:00.533621 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.533540    2571 scope.go:117] "RemoveContainer" containerID="265690aa462eb7f08d16c1e39ee0687a04c2e71e7fc2d403053242e6a4b0ad25"
Jul 28 18:24:00.540305 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.540286    2571 scope.go:117] "RemoveContainer" containerID="d13f8ac2687a852113512f2d93cef2b51e70930c0cba3bcd6b6ae6280e9005f2"
Jul 28 18:24:00.548115 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.548096    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"]
Jul 28 18:24:00.554370 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:00.554352    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-5hwzp"]
Jul 28 18:24:01.706636 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:24:01.706611    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" path="/var/lib/kubelet/pods/438fa904-0ae8-4f8c-95d3-574067d4e5c5/volumes"
Jul 28 18:29:17.472289 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472240    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"]
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472480    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472489    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472498    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472503    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472516    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472521    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472529    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="storage-initializer"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472534    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="storage-initializer"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472543    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472548    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472557    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="storage-initializer"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472561    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="storage-initializer"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472596    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kube-rbac-proxy"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472605    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kserve-container"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472611    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f2db8f-1613-45ea-a5d4-3a79ecdafb83" containerName="kube-rbac-proxy"
Jul 28 18:29:17.472734 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.472618    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="438fa904-0ae8-4f8c-95d3-574067d4e5c5" containerName="kserve-container"
Jul 28 18:29:17.474516 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.474501    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.477430 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.477407    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:29:17.478422 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.478395    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:29:17.478422 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.478409    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:29:17.478422 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.478413    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:29:17.478633 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.478462    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:29:17.478633 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.478469    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 18:29:17.478633 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.478474    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:29:17.486822 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.486805    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"]
Jul 28 18:29:17.613211 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.613186    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8521dba-d375-4244-ba84-bed7752e550e-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.613330 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.613228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8521dba-d375-4244-ba84-bed7752e550e-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.613330 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.613272    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4wnc\" (UniqueName: \"kubernetes.io/projected/e8521dba-d375-4244-ba84-bed7752e550e-kube-api-access-j4wnc\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.613330 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.613297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.613330 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.613311    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714304 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714281    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8521dba-d375-4244-ba84-bed7752e550e-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4wnc\" (UniqueName: \"kubernetes.io/projected/e8521dba-d375-4244-ba84-bed7752e550e-kube-api-access-j4wnc\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714345    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714424 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714414    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8521dba-d375-4244-ba84-bed7752e550e-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714696 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714678    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8521dba-d375-4244-ba84-bed7752e550e-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.714994 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714975    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.715043 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.714976    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.716710 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.716690    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8521dba-d375-4244-ba84-bed7752e550e-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.723911 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.723862    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4wnc\" (UniqueName: \"kubernetes.io/projected/e8521dba-d375-4244-ba84-bed7752e550e-kube-api-access-j4wnc\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-z5llr\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.783752 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.783730    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:17.910457 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:17.910346    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"]
Jul 28 18:29:17.913045 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:29:17.913020    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8521dba_d375_4244_ba84_bed7752e550e.slice/crio-5bbb8008b4b9710d05b359e0bf65464085335944bbf987d9c73a04e54d0dc12c WatchSource:0}: Error finding container 5bbb8008b4b9710d05b359e0bf65464085335944bbf987d9c73a04e54d0dc12c: Status 404 returned error can't find the container with id 5bbb8008b4b9710d05b359e0bf65464085335944bbf987d9c73a04e54d0dc12c
Jul 28 18:29:18.501494 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:18.501461    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerStarted","Data":"13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c"}
Jul 28 18:29:18.501971 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:18.501521    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerStarted","Data":"5bbb8008b4b9710d05b359e0bf65464085335944bbf987d9c73a04e54d0dc12c"}
Jul 28 18:29:22.515774 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:22.515745    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8521dba-d375-4244-ba84-bed7752e550e" containerID="13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c" exitCode=0
Jul 28 18:29:22.516117 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:22.515807    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerDied","Data":"13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c"}
Jul 28 18:29:22.516974 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:22.516959    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:29:40.590773 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:40.590740    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerStarted","Data":"3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1"}
Jul 28 18:29:40.591126 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:40.590795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerStarted","Data":"eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f"}
Jul 28 18:29:40.591126 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:40.591122    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:40.591286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:40.591268    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:40.592397 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:40.592373    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:29:40.612789 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:40.612750    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podStartSLOduration=2.365721326 podStartE2EDuration="23.612739136s" totalImagesPullingTime="17.462312386s" totalInitContainerRuntime="3.784705424s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:40.611392229 +0000 UTC m=+4765.371417856" watchObservedRunningTime="2026-07-28 18:29:40.612739136 +0000 UTC m=+4765.372764831"
Jul 28 18:29:41.594299 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:41.594266    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:29:42.596876 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:42.596838    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:29:47.601271 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:47.601224    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:29:47.601730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:47.601702    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:29:57.602597 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:29:57.602559    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:30:07.601892 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:07.601846    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:30:17.601735 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:17.601694    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:30:27.602266 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:27.602212    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 28 18:30:37.603004 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:37.602972    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:30:47.618925 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:47.618894    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"]
Jul 28 18:30:47.619393 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:47.619194    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container" containerID="cri-o://eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f" gracePeriod=30
Jul 28 18:30:47.619393 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:47.619219    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kube-rbac-proxy" containerID="cri-o://3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1" gracePeriod=30
Jul 28 18:30:47.816525 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:47.816499    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8521dba-d375-4244-ba84-bed7752e550e" containerID="3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1" exitCode=2
Jul 28 18:30:47.816626 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:47.816568    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerDied","Data":"3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1"}
Jul 28 18:30:50.357601 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.357579    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:30:50.443477 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443418    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-cabundle-cert\") pod \"e8521dba-d375-4244-ba84-bed7752e550e\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") "
Jul 28 18:30:50.443477 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443458    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4wnc\" (UniqueName: \"kubernetes.io/projected/e8521dba-d375-4244-ba84-bed7752e550e-kube-api-access-j4wnc\") pod \"e8521dba-d375-4244-ba84-bed7752e550e\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") "
Jul 28 18:30:50.443621 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443499    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8521dba-d375-4244-ba84-bed7752e550e-kserve-provision-location\") pod \"e8521dba-d375-4244-ba84-bed7752e550e\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") "
Jul 28 18:30:50.443621 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443525    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8521dba-d375-4244-ba84-bed7752e550e-proxy-tls\") pod \"e8521dba-d375-4244-ba84-bed7752e550e\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") "
Jul 28 18:30:50.443621 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443539    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-kube-rbac-proxy-sar-config\") pod \"e8521dba-d375-4244-ba84-bed7752e550e\" (UID: \"e8521dba-d375-4244-ba84-bed7752e550e\") "
Jul 28 18:30:50.443874 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443786    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8521dba-d375-4244-ba84-bed7752e550e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8521dba-d375-4244-ba84-bed7752e550e" (UID: "e8521dba-d375-4244-ba84-bed7752e550e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:50.443942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443799    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8521dba-d375-4244-ba84-bed7752e550e" (UID: "e8521dba-d375-4244-ba84-bed7752e550e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:50.443995 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.443947    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8521dba-d375-4244-ba84-bed7752e550e" (UID: "e8521dba-d375-4244-ba84-bed7752e550e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:50.445592 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.445569    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8521dba-d375-4244-ba84-bed7752e550e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8521dba-d375-4244-ba84-bed7752e550e" (UID: "e8521dba-d375-4244-ba84-bed7752e550e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:50.445592 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.445581    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8521dba-d375-4244-ba84-bed7752e550e-kube-api-access-j4wnc" (OuterVolumeSpecName: "kube-api-access-j4wnc") pod "e8521dba-d375-4244-ba84-bed7752e550e" (UID: "e8521dba-d375-4244-ba84-bed7752e550e"). InnerVolumeSpecName "kube-api-access-j4wnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:50.544792 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.544763    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8521dba-d375-4244-ba84-bed7752e550e-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:30:50.544792 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.544792    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8521dba-d375-4244-ba84-bed7752e550e-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:30:50.544913 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.544803    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:30:50.544913 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.544811    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8521dba-d375-4244-ba84-bed7752e550e-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:30:50.544913 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.544819    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4wnc\" (UniqueName: \"kubernetes.io/projected/e8521dba-d375-4244-ba84-bed7752e550e-kube-api-access-j4wnc\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:30:50.827531 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.827475    2571 generic.go:356] "Generic (PLEG): container finished" podID="e8521dba-d375-4244-ba84-bed7752e550e" containerID="eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f" exitCode=0
Jul 28 18:30:50.827531 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.827525    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"
Jul 28 18:30:50.827686 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.827558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerDied","Data":"eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f"}
Jul 28 18:30:50.827686 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.827617    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr" event={"ID":"e8521dba-d375-4244-ba84-bed7752e550e","Type":"ContainerDied","Data":"5bbb8008b4b9710d05b359e0bf65464085335944bbf987d9c73a04e54d0dc12c"}
Jul 28 18:30:50.827686 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.827653    2571 scope.go:117] "RemoveContainer" containerID="3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1"
Jul 28 18:30:50.835924 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.835904    2571 scope.go:117] "RemoveContainer" containerID="eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f"
Jul 28 18:30:50.842298 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.842277    2571 scope.go:117] "RemoveContainer" containerID="13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c"
Jul 28 18:30:50.848576 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.848561    2571 scope.go:117] "RemoveContainer" containerID="3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1"
Jul 28 18:30:50.848816 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:30:50.848800    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1\": container with ID starting with 3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1 not found: ID does not exist" containerID="3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1"
Jul 28 18:30:50.848857 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.848824    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1"} err="failed to get container status \"3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1\": rpc error: code = NotFound desc = could not find container \"3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1\": container with ID starting with 3eed6a446dbc455972179c1706a335278ffd561f11a73f0e23a32213e53734d1 not found: ID does not exist"
Jul 28 18:30:50.848857 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.848839    2571 scope.go:117] "RemoveContainer" containerID="eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f"
Jul 28 18:30:50.849102 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:30:50.849078    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f\": container with ID starting with eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f not found: ID does not exist" containerID="eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f"
Jul 28 18:30:50.849177 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.849110    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f"} err="failed to get container status \"eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f\": rpc error: code = NotFound desc = could not find container \"eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f\": container with ID starting with eb761e0b7f10bfcbddbf119a29250d5f71837fafb1e168a73796ea344bf82f8f not found: ID does not exist"
Jul 28 18:30:50.849177 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.849140    2571 scope.go:117] "RemoveContainer" containerID="13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c"
Jul 28 18:30:50.849556 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:30:50.849523    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c\": container with ID starting with 13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c not found: ID does not exist" containerID="13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c"
Jul 28 18:30:50.849656 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.849563    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c"} err="failed to get container status \"13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c\": rpc error: code = NotFound desc = could not find container \"13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c\": container with ID starting with 13a5ac973eca2edfab9bbae7b0def4f565a241b6590d70d0738bdce08df0933c not found: ID does not exist"
Jul 28 18:30:50.850708 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.850692    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"]
Jul 28 18:30:50.854983 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:50.854963    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-z5llr"]
Jul 28 18:30:51.707196 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:30:51.707167    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8521dba-d375-4244-ba84-bed7752e550e" path="/var/lib/kubelet/pods/e8521dba-d375-4244-ba84-bed7752e550e/volumes"
Jul 28 18:31:58.127510 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127474    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"]
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127696    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="storage-initializer"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127706    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="storage-initializer"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127714    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kube-rbac-proxy"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127720    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kube-rbac-proxy"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127731    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127738    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127778    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kserve-container"
Jul 28 18:31:58.127942 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.127787    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8521dba-d375-4244-ba84-bed7752e550e" containerName="kube-rbac-proxy"
Jul 28 18:31:58.129535 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.129520    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.132977 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.132957    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 18:31:58.133101 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.132978    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:31:58.133267 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.133234    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:31:58.133321 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.133280    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:31:58.134313 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.134297    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:31:58.134436 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.134298    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:31:58.134436 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.134329    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:31:58.155906 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.155885    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"]
Jul 28 18:31:58.174427 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.174407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15efac17-7e5b-4da8-872a-f23dc7723639-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.174538 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.174444    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsjm\" (UniqueName: \"kubernetes.io/projected/15efac17-7e5b-4da8-872a-f23dc7723639-kube-api-access-sgsjm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.174538 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.174489    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15efac17-7e5b-4da8-872a-f23dc7723639-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.174538 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.174514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.174688 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.174604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275014 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.274988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275115 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15efac17-7e5b-4da8-872a-f23dc7723639-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275115 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275080    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsjm\" (UniqueName: \"kubernetes.io/projected/15efac17-7e5b-4da8-872a-f23dc7723639-kube-api-access-sgsjm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15efac17-7e5b-4da8-872a-f23dc7723639-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275443 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275426    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15efac17-7e5b-4da8-872a-f23dc7723639-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275726 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275701    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.275726 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.275716    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.277466 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.277447    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15efac17-7e5b-4da8-872a-f23dc7723639-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.284175 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.284153    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsjm\" (UniqueName: \"kubernetes.io/projected/15efac17-7e5b-4da8-872a-f23dc7723639-kube-api-access-sgsjm\") pod \"isvc-xgboost-v2-predictor-fcc668bc-pw4qx\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.439098 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.439074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:31:58.560348 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:58.560315    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"]
Jul 28 18:31:58.562956 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:31:58.562928    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15efac17_7e5b_4da8_872a_f23dc7723639.slice/crio-3c73fc8d62a1d30f67481e11cb3c6fcd2a908f78b586df0d3ec514a5a5fe451f WatchSource:0}: Error finding container 3c73fc8d62a1d30f67481e11cb3c6fcd2a908f78b586df0d3ec514a5a5fe451f: Status 404 returned error can't find the container with id 3c73fc8d62a1d30f67481e11cb3c6fcd2a908f78b586df0d3ec514a5a5fe451f
Jul 28 18:31:59.045719 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:59.045686    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerStarted","Data":"800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a"}
Jul 28 18:31:59.045877 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:31:59.045746    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerStarted","Data":"3c73fc8d62a1d30f67481e11cb3c6fcd2a908f78b586df0d3ec514a5a5fe451f"}
Jul 28 18:32:03.059304 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:03.059272    2571 generic.go:356] "Generic (PLEG): container finished" podID="15efac17-7e5b-4da8-872a-f23dc7723639" containerID="800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a" exitCode=0
Jul 28 18:32:03.059675 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:03.059343    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerDied","Data":"800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a"}
Jul 28 18:32:04.065200 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:04.065170    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerStarted","Data":"dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd"}
Jul 28 18:32:04.065547 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:04.065238    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerStarted","Data":"d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be"}
Jul 28 18:32:04.065547 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:04.065425    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:32:04.084884 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:04.084842    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podStartSLOduration=2.421873491 podStartE2EDuration="6.084831438s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662957947s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:32:04.084075778 +0000 UTC m=+4908.844101416" watchObservedRunningTime="2026-07-28 18:32:04.084831438 +0000 UTC m=+4908.844857064"
Jul 28 18:32:05.068212 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:05.068183    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:32:05.069156 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:05.069127    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:32:06.071038 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:06.071000    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:32:11.075023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:11.074996    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:32:11.075506 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:11.075477    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:32:21.076171 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:21.076132    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:32:31.075926 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:31.075889    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:32:41.075581 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:41.075497    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:32:51.075977 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:32:51.075940    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 28 18:33:01.076159 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:01.076126    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:33:08.208126 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:08.208091    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"]
Jul 28 18:33:08.208699 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:08.208376    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container" containerID="cri-o://d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be" gracePeriod=30
Jul 28 18:33:08.208699 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:08.208673    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kube-rbac-proxy" containerID="cri-o://dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd" gracePeriod=30
Jul 28 18:33:09.278445 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:09.278410    2571 generic.go:356] "Generic (PLEG): container finished" podID="15efac17-7e5b-4da8-872a-f23dc7723639" containerID="dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd" exitCode=2
Jul 28 18:33:09.278805 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:09.278468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerDied","Data":"dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd"}
Jul 28 18:33:10.937426 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:10.937404    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:33:11.045704 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.045645    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15efac17-7e5b-4da8-872a-f23dc7723639-proxy-tls\") pod \"15efac17-7e5b-4da8-872a-f23dc7723639\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") "
Jul 28 18:33:11.045704 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.045685    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15efac17-7e5b-4da8-872a-f23dc7723639-kserve-provision-location\") pod \"15efac17-7e5b-4da8-872a-f23dc7723639\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") "
Jul 28 18:33:11.045885 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.045713    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgsjm\" (UniqueName: \"kubernetes.io/projected/15efac17-7e5b-4da8-872a-f23dc7723639-kube-api-access-sgsjm\") pod \"15efac17-7e5b-4da8-872a-f23dc7723639\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") "
Jul 28 18:33:11.045885 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.045762    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-kube-rbac-proxy-sar-config\") pod \"15efac17-7e5b-4da8-872a-f23dc7723639\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") "
Jul 28 18:33:11.045885 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.045793    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-cabundle-cert\") pod \"15efac17-7e5b-4da8-872a-f23dc7723639\" (UID: \"15efac17-7e5b-4da8-872a-f23dc7723639\") "
Jul 28 18:33:11.046023 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.045958    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15efac17-7e5b-4da8-872a-f23dc7723639-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15efac17-7e5b-4da8-872a-f23dc7723639" (UID: "15efac17-7e5b-4da8-872a-f23dc7723639"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:11.046113 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.046086    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15efac17-7e5b-4da8-872a-f23dc7723639" (UID: "15efac17-7e5b-4da8-872a-f23dc7723639"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:33:11.046165 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.046151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15efac17-7e5b-4da8-872a-f23dc7723639" (UID: "15efac17-7e5b-4da8-872a-f23dc7723639"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:33:11.047657 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.047631    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15efac17-7e5b-4da8-872a-f23dc7723639-kube-api-access-sgsjm" (OuterVolumeSpecName: "kube-api-access-sgsjm") pod "15efac17-7e5b-4da8-872a-f23dc7723639" (UID: "15efac17-7e5b-4da8-872a-f23dc7723639"). InnerVolumeSpecName "kube-api-access-sgsjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:33:11.047746 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.047640    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15efac17-7e5b-4da8-872a-f23dc7723639-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15efac17-7e5b-4da8-872a-f23dc7723639" (UID: "15efac17-7e5b-4da8-872a-f23dc7723639"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:33:11.147046 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.147025    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-cabundle-cert\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:33:11.147046 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.147044    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15efac17-7e5b-4da8-872a-f23dc7723639-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:33:11.147163 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.147054    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15efac17-7e5b-4da8-872a-f23dc7723639-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:33:11.147163 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.147063    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsjm\" (UniqueName: \"kubernetes.io/projected/15efac17-7e5b-4da8-872a-f23dc7723639-kube-api-access-sgsjm\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:33:11.147163 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.147071    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15efac17-7e5b-4da8-872a-f23dc7723639-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:33:11.286546 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.286524    2571 generic.go:356] "Generic (PLEG): container finished" podID="15efac17-7e5b-4da8-872a-f23dc7723639" containerID="d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be" exitCode=0
Jul 28 18:33:11.286632 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.286580    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"
Jul 28 18:33:11.286632 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.286610    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerDied","Data":"d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be"}
Jul 28 18:33:11.286705 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.286674    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx" event={"ID":"15efac17-7e5b-4da8-872a-f23dc7723639","Type":"ContainerDied","Data":"3c73fc8d62a1d30f67481e11cb3c6fcd2a908f78b586df0d3ec514a5a5fe451f"}
Jul 28 18:33:11.286741 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.286712    2571 scope.go:117] "RemoveContainer" containerID="dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd"
Jul 28 18:33:11.294874 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.294856    2571 scope.go:117] "RemoveContainer" containerID="d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be"
Jul 28 18:33:11.301602 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.301588    2571 scope.go:117] "RemoveContainer" containerID="800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a"
Jul 28 18:33:11.310116 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.310094    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"]
Jul 28 18:33:11.310829 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.310815    2571 scope.go:117] "RemoveContainer" containerID="dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd"
Jul 28 18:33:11.311096 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:33:11.311078    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd\": container with ID starting with dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd not found: ID does not exist" containerID="dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd"
Jul 28 18:33:11.311153 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.311103    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd"} err="failed to get container status \"dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd\": rpc error: code = NotFound desc = could not find container \"dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd\": container with ID starting with dd8891f4c03bb66a07d4d6e21b22f80d1ab192d943122c157ba51eb66448c3dd not found: ID does not exist"
Jul 28 18:33:11.311153 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.311119    2571 scope.go:117] "RemoveContainer" containerID="d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be"
Jul 28 18:33:11.311499 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:33:11.311446    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be\": container with ID starting with d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be not found: ID does not exist" containerID="d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be"
Jul 28 18:33:11.311499 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.311479    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be"} err="failed to get container status \"d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be\": rpc error: code = NotFound desc = could not find container \"d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be\": container with ID starting with d2af8f99c2e5ba4864cd90275f225605177d33e4649b33e2f34931e776b404be not found: ID does not exist"
Jul 28 18:33:11.311637 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.311502    2571 scope.go:117] "RemoveContainer" containerID="800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a"
Jul 28 18:33:11.311976 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:33:11.311951    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a\": container with ID starting with 800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a not found: ID does not exist" containerID="800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a"
Jul 28 18:33:11.312050 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.311980    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a"} err="failed to get container status \"800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a\": rpc error: code = NotFound desc = could not find container \"800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a\": container with ID starting with 800c9d911518ba0e6c14cdea4bea8f199023338eaf8cc3d0e5929b3f72fb037a not found: ID does not exist"
Jul 28 18:33:11.313115 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.313098    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-pw4qx"]
Jul 28 18:33:11.706465 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:11.706441    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" path="/var/lib/kubelet/pods/15efac17-7e5b-4da8-872a-f23dc7723639/volumes"
Jul 28 18:33:13.379909 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.379878    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"]
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380108    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kube-rbac-proxy"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380118    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kube-rbac-proxy"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380135    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="storage-initializer"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380141    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="storage-initializer"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380149    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380155    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380192    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kserve-container"
Jul 28 18:33:13.380270 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.380198    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="15efac17-7e5b-4da8-872a-f23dc7723639" containerName="kube-rbac-proxy"
Jul 28 18:33:13.383305 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.383290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.385764 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.385743    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:33:13.385873 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.385805    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:33:13.386032 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.386017    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 18:33:13.386091 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.386037    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 18:33:13.387160 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.387144    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:33:13.387279 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.387183    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:33:13.393141 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.393120    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"]
Jul 28 18:33:13.460013 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.459987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8ck\" (UniqueName: \"kubernetes.io/projected/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-api-access-nv8ck\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.460123 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.460026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.460123 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.460057    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.460123 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.460086    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.560597 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.560574    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8ck\" (UniqueName: \"kubernetes.io/projected/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-api-access-nv8ck\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.560680 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.560600    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.560680 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.560620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.560680 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.560639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.560953 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.560937    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.561293 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.561275    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.562959 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.562939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.568476 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.568457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8ck\" (UniqueName: \"kubernetes.io/projected/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-api-access-nv8ck\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.693050 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.693026    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:13.814987 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:13.814958    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"]
Jul 28 18:33:13.818395 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:33:13.818372    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0375a2e4_e2ee_4290_a677_0ddf4f3d7d02.slice/crio-a7176061be7b1b13da49fff04d491d142ab8792af3407e147762dce028dc51c5 WatchSource:0}: Error finding container a7176061be7b1b13da49fff04d491d142ab8792af3407e147762dce028dc51c5: Status 404 returned error can't find the container with id a7176061be7b1b13da49fff04d491d142ab8792af3407e147762dce028dc51c5
Jul 28 18:33:14.300318 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:14.300285    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerStarted","Data":"0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52"}
Jul 28 18:33:14.300486 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:14.300339    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerStarted","Data":"a7176061be7b1b13da49fff04d491d142ab8792af3407e147762dce028dc51c5"}
Jul 28 18:33:15.310137 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:15.310107    2571 generic.go:356] "Generic (PLEG): container finished" podID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerID="0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52" exitCode=0
Jul 28 18:33:15.310537 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:15.310161    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerDied","Data":"0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52"}
Jul 28 18:33:16.315369 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:16.315329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerStarted","Data":"00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78"}
Jul 28 18:33:16.315803 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:16.315393    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:16.315803 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:16.315421    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerStarted","Data":"636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4"}
Jul 28 18:33:16.315803 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:16.315450    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:16.316600 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:16.316573    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:33:16.334762 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:16.334717    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podStartSLOduration=2.386031635 podStartE2EDuration="3.334705557s" totalImagesPullingTime="0s" totalInitContainerRuntime="948.673922ms" isStatefulPod=false podCreationTimestamp="2026-07-28 18:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:16.33448988 +0000 UTC m=+4981.094515525" watchObservedRunningTime="2026-07-28 18:33:16.334705557 +0000 UTC m=+4981.094731183"
Jul 28 18:33:17.319271 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:17.319223    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:33:22.323731 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:22.323701    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:33:22.324206 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:22.324180    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:33:32.325059 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:32.325023    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:33:42.324359 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:42.324321    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:33:52.324847 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:33:52.324808    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:34:02.324606 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:02.324567    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:34:12.324343 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:12.324268    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 28 18:34:22.324402 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:22.324371    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:34:23.477760 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:23.477728    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"]
Jul 28 18:34:23.478194 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:23.478002    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container" containerID="cri-o://636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4" gracePeriod=30
Jul 28 18:34:23.478194 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:23.478024    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kube-rbac-proxy" containerID="cri-o://00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78" gracePeriod=30
Jul 28 18:34:24.538037 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:24.538002    2571 generic.go:356] "Generic (PLEG): container finished" podID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerID="00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78" exitCode=2
Jul 28 18:34:24.538422 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:24.538076    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerDied","Data":"00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78"}
Jul 28 18:34:27.218790 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.218765    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:34:27.227947 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.227930    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kserve-provision-location\") pod \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") "
Jul 28 18:34:27.228014 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.227957    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv8ck\" (UniqueName: \"kubernetes.io/projected/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-api-access-nv8ck\") pod \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") "
Jul 28 18:34:27.228014 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.227981    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-proxy-tls\") pod \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") "
Jul 28 18:34:27.228014 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.228006    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-rbac-proxy-sar-config\") pod \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\" (UID: \"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02\") "
Jul 28 18:34:27.228310 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.228282    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" (UID: "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:27.228368 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.228307    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" (UID: "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:34:27.229844 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.229819    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" (UID: "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:27.229945 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.229906    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-api-access-nv8ck" (OuterVolumeSpecName: "kube-api-access-nv8ck") pod "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" (UID: "0375a2e4-e2ee-4290-a677-0ddf4f3d7d02"). InnerVolumeSpecName "kube-api-access-nv8ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:27.329312 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.329261    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:34:27.329312 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.329281    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv8ck\" (UniqueName: \"kubernetes.io/projected/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-api-access-nv8ck\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:34:27.329312 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.329290    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:34:27.329312 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.329298    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:34:27.548968 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.548941    2571 generic.go:356] "Generic (PLEG): container finished" podID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerID="636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4" exitCode=0
Jul 28 18:34:27.549068 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.548990    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"
Jul 28 18:34:27.549068 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.549014    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerDied","Data":"636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4"}
Jul 28 18:34:27.549068 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.549064    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk" event={"ID":"0375a2e4-e2ee-4290-a677-0ddf4f3d7d02","Type":"ContainerDied","Data":"a7176061be7b1b13da49fff04d491d142ab8792af3407e147762dce028dc51c5"}
Jul 28 18:34:27.549175 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.549091    2571 scope.go:117] "RemoveContainer" containerID="00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78"
Jul 28 18:34:27.556807 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.556778    2571 scope.go:117] "RemoveContainer" containerID="636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4"
Jul 28 18:34:27.563475 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.563460    2571 scope.go:117] "RemoveContainer" containerID="0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52"
Jul 28 18:34:27.567912 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.567892    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"]
Jul 28 18:34:27.570134 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.570117    2571 scope.go:117] "RemoveContainer" containerID="00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78"
Jul 28 18:34:27.570572 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:34:27.570439    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78\": container with ID starting with 00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78 not found: ID does not exist" containerID="00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78"
Jul 28 18:34:27.570572 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.570475    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78"} err="failed to get container status \"00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78\": rpc error: code = NotFound desc = could not find container \"00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78\": container with ID starting with 00298fed082a4af6a0c984e033fd675aa28435124871f2a05890128743a91c78 not found: ID does not exist"
Jul 28 18:34:27.570572 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.570499    2571 scope.go:117] "RemoveContainer" containerID="636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4"
Jul 28 18:34:27.570781 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:34:27.570760    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4\": container with ID starting with 636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4 not found: ID does not exist" containerID="636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4"
Jul 28 18:34:27.570820 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.570790    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4"} err="failed to get container status \"636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4\": rpc error: code = NotFound desc = could not find container \"636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4\": container with ID starting with 636babc557ab699c9dd342096e88f44d7ccf8379949c8ce222621af28b063de4 not found: ID does not exist"
Jul 28 18:34:27.570820 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.570813    2571 scope.go:117] "RemoveContainer" containerID="0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52"
Jul 28 18:34:27.571101 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:34:27.571081    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52\": container with ID starting with 0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52 not found: ID does not exist" containerID="0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52"
Jul 28 18:34:27.571171 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.571107    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52"} err="failed to get container status \"0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52\": rpc error: code = NotFound desc = could not find container \"0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52\": container with ID starting with 0c64b46d8041550ecc7639e8285f80ac08354f0172ee6bad5bef4e196097ad52 not found: ID does not exist"
Jul 28 18:34:27.571718 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.571699    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-vgwtk"]
Jul 28 18:34:27.710788 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:34:27.708093    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" path="/var/lib/kubelet/pods/0375a2e4-e2ee-4290-a677-0ddf4f3d7d02/volumes"
Jul 28 18:35:44.760356 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760324    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"]
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760545    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760554    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760568    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kube-rbac-proxy"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760574    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kube-rbac-proxy"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760582    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="storage-initializer"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760588    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="storage-initializer"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760626    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kserve-container"
Jul 28 18:35:44.760730 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.760635    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0375a2e4-e2ee-4290-a677-0ddf4f3d7d02" containerName="kube-rbac-proxy"
Jul 28 18:35:44.763437 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.763422    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.766131 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.766104    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:35:44.766131 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.766129    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:35:44.766329 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.766108    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 18:35:44.766469 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.766456    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:35:44.766628 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.766612    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:35:44.767328 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.767305    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:35:44.773276 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.773234    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"]
Jul 28 18:35:44.833040 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.833015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef5adb9c-a425-4517-80c6-f73dcd484239-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.833151 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.833044    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.833195 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.833164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef5adb9c-a425-4517-80c6-f73dcd484239-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.833195 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.833188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrgq\" (UniqueName: \"kubernetes.io/projected/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-api-access-wvrgq\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.933535 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.933512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef5adb9c-a425-4517-80c6-f73dcd484239-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.933645 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.933537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrgq\" (UniqueName: \"kubernetes.io/projected/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-api-access-wvrgq\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.933645 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.933559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef5adb9c-a425-4517-80c6-f73dcd484239-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.933645 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.933575    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.933971 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.933952    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef5adb9c-a425-4517-80c6-f73dcd484239-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.934224 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.934209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.935838 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.935822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef5adb9c-a425-4517-80c6-f73dcd484239-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:44.942075 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:44.942057    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrgq\" (UniqueName: \"kubernetes.io/projected/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-api-access-wvrgq\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:45.073722 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:45.073663    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:45.195722 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:45.195609    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"]
Jul 28 18:35:45.198211 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:35:45.198182    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5adb9c_a425_4517_80c6_f73dcd484239.slice/crio-07f28603dda465c22579c0dbbdfc3a45e24ac80b855aaab63d4701f3aa79abb2 WatchSource:0}: Error finding container 07f28603dda465c22579c0dbbdfc3a45e24ac80b855aaab63d4701f3aa79abb2: Status 404 returned error can't find the container with id 07f28603dda465c22579c0dbbdfc3a45e24ac80b855aaab63d4701f3aa79abb2
Jul 28 18:35:45.794740 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:45.794707    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" event={"ID":"ef5adb9c-a425-4517-80c6-f73dcd484239","Type":"ContainerStarted","Data":"023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c"}
Jul 28 18:35:45.795094 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:45.794762    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" event={"ID":"ef5adb9c-a425-4517-80c6-f73dcd484239","Type":"ContainerStarted","Data":"07f28603dda465c22579c0dbbdfc3a45e24ac80b855aaab63d4701f3aa79abb2"}
Jul 28 18:35:51.815354 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:51.815332    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn_ef5adb9c-a425-4517-80c6-f73dcd484239/storage-initializer/0.log"
Jul 28 18:35:51.815802 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:51.815366    2571 generic.go:356] "Generic (PLEG): container finished" podID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerID="023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c" exitCode=1
Jul 28 18:35:51.815802 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:51.815414    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" event={"ID":"ef5adb9c-a425-4517-80c6-f73dcd484239","Type":"ContainerDied","Data":"023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c"}
Jul 28 18:35:52.820922 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:52.820898    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn_ef5adb9c-a425-4517-80c6-f73dcd484239/storage-initializer/0.log"
Jul 28 18:35:52.821323 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:52.820982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" event={"ID":"ef5adb9c-a425-4517-80c6-f73dcd484239","Type":"ContainerStarted","Data":"0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c"}
Jul 28 18:35:54.781259 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:54.781218    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"]
Jul 28 18:35:54.781620 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:54.781422    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer" containerID="cri-o://0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c" gracePeriod=30
Jul 28 18:35:57.407386 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.407363    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn_ef5adb9c-a425-4517-80c6-f73dcd484239/storage-initializer/1.log"
Jul 28 18:35:57.407840 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.407824    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn_ef5adb9c-a425-4517-80c6-f73dcd484239/storage-initializer/0.log"
Jul 28 18:35:57.410348 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.410321    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:57.515285 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.515202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvrgq\" (UniqueName: \"kubernetes.io/projected/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-api-access-wvrgq\") pod \"ef5adb9c-a425-4517-80c6-f73dcd484239\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") "
Jul 28 18:35:57.515285 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.515270    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-rbac-proxy-sar-config\") pod \"ef5adb9c-a425-4517-80c6-f73dcd484239\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") "
Jul 28 18:35:57.515450 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.515356    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef5adb9c-a425-4517-80c6-f73dcd484239-kserve-provision-location\") pod \"ef5adb9c-a425-4517-80c6-f73dcd484239\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") "
Jul 28 18:35:57.515450 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.515389    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef5adb9c-a425-4517-80c6-f73dcd484239-proxy-tls\") pod \"ef5adb9c-a425-4517-80c6-f73dcd484239\" (UID: \"ef5adb9c-a425-4517-80c6-f73dcd484239\") "
Jul 28 18:35:57.515641 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.515616    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5adb9c-a425-4517-80c6-f73dcd484239-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef5adb9c-a425-4517-80c6-f73dcd484239" (UID: "ef5adb9c-a425-4517-80c6-f73dcd484239"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:35:57.515703 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.515634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef5adb9c-a425-4517-80c6-f73dcd484239" (UID: "ef5adb9c-a425-4517-80c6-f73dcd484239"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:35:57.517193 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.517165    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-api-access-wvrgq" (OuterVolumeSpecName: "kube-api-access-wvrgq") pod "ef5adb9c-a425-4517-80c6-f73dcd484239" (UID: "ef5adb9c-a425-4517-80c6-f73dcd484239"). InnerVolumeSpecName "kube-api-access-wvrgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:35:57.517311 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.517239    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5adb9c-a425-4517-80c6-f73dcd484239-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef5adb9c-a425-4517-80c6-f73dcd484239" (UID: "ef5adb9c-a425-4517-80c6-f73dcd484239"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:35:57.616243 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.616218    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:35:57.616243 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.616238    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef5adb9c-a425-4517-80c6-f73dcd484239-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:35:57.616377 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.616265    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef5adb9c-a425-4517-80c6-f73dcd484239-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:35:57.616377 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.616275    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrgq\" (UniqueName: \"kubernetes.io/projected/ef5adb9c-a425-4517-80c6-f73dcd484239-kube-api-access-wvrgq\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:35:57.839279 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839232    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn_ef5adb9c-a425-4517-80c6-f73dcd484239/storage-initializer/1.log"
Jul 28 18:35:57.839631 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839617    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn_ef5adb9c-a425-4517-80c6-f73dcd484239/storage-initializer/0.log"
Jul 28 18:35:57.839682 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839649    2571 generic.go:356] "Generic (PLEG): container finished" podID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerID="0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c" exitCode=1
Jul 28 18:35:57.839717 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839691    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" event={"ID":"ef5adb9c-a425-4517-80c6-f73dcd484239","Type":"ContainerDied","Data":"0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c"}
Jul 28 18:35:57.839717 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839702    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"
Jul 28 18:35:57.839783 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn" event={"ID":"ef5adb9c-a425-4517-80c6-f73dcd484239","Type":"ContainerDied","Data":"07f28603dda465c22579c0dbbdfc3a45e24ac80b855aaab63d4701f3aa79abb2"}
Jul 28 18:35:57.839783 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.839754    2571 scope.go:117] "RemoveContainer" containerID="0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c"
Jul 28 18:35:57.847368 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.847346    2571 scope.go:117] "RemoveContainer" containerID="023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c"
Jul 28 18:35:57.853800 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.853785    2571 scope.go:117] "RemoveContainer" containerID="0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c"
Jul 28 18:35:57.854025 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:35:57.854006    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c\": container with ID starting with 0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c not found: ID does not exist" containerID="0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c"
Jul 28 18:35:57.854076 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.854032    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c"} err="failed to get container status \"0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c\": rpc error: code = NotFound desc = could not find container \"0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c\": container with ID starting with 0d6a425a11df607dccdab289ddda03aedfd3065dfa81fc95ecae1b4149e1777c not found: ID does not exist"
Jul 28 18:35:57.854076 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.854050    2571 scope.go:117] "RemoveContainer" containerID="023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c"
Jul 28 18:35:57.854304 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:35:57.854279    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c\": container with ID starting with 023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c not found: ID does not exist" containerID="023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c"
Jul 28 18:35:57.854394 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.854305    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c"} err="failed to get container status \"023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c\": rpc error: code = NotFound desc = could not find container \"023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c\": container with ID starting with 023dfd47b4a29be7562ac7c68ff83634f9ddf15c7d6ceb4445e6e19859736d2c not found: ID does not exist"
Jul 28 18:35:57.872973 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.872951    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"]
Jul 28 18:35:57.876573 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:57.876554    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-wgvnn"]
Jul 28 18:35:59.707038 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:35:59.706999    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" path="/var/lib/kubelet/pods/ef5adb9c-a425-4517-80c6-f73dcd484239/volumes"
Jul 28 18:37:17.020177 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020109    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"]
Jul 28 18:37:17.020619 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020386    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer"
Jul 28 18:37:17.020619 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020398    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer"
Jul 28 18:37:17.020619 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020448    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer"
Jul 28 18:37:17.020619 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020457    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer"
Jul 28 18:37:17.020619 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020501    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer"
Jul 28 18:37:17.020619 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.020507    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5adb9c-a425-4517-80c6-f73dcd484239" containerName="storage-initializer"
Jul 28 18:37:17.023364 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.023345    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.029961 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.029942    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:37:17.030095 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.029981    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 18:37:17.030095 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.029989    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:37:17.030267 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.030215    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:37:17.031008 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.030994    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:37:17.042812 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.042795    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"]
Jul 28 18:37:17.049957 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.049938    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:37:17.123758 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.123736    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cw2m\" (UniqueName: \"kubernetes.io/projected/f8405899-6d85-4780-b49c-c403654373cb-kube-api-access-8cw2m\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.123845 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.123763    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.123845 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.123780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8405899-6d85-4780-b49c-c403654373cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.123845 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.123838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8405899-6d85-4780-b49c-c403654373cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.224459 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.224437    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8405899-6d85-4780-b49c-c403654373cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.224535 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.224485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cw2m\" (UniqueName: \"kubernetes.io/projected/f8405899-6d85-4780-b49c-c403654373cb-kube-api-access-8cw2m\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.224535 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.224503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.224535 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.224522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8405899-6d85-4780-b49c-c403654373cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.224651 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:37:17.224634    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 28 18:37:17.224727 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:37:17.224713    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls podName:f8405899-6d85-4780-b49c-c403654373cb nodeName:}" failed. No retries permitted until 2026-07-28 18:37:17.724695989 +0000 UTC m=+5222.484721605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" (UID: "f8405899-6d85-4780-b49c-c403654373cb") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 28 18:37:17.224788 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.224754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8405899-6d85-4780-b49c-c403654373cb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.225084 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.225065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8405899-6d85-4780-b49c-c403654373cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.233614 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.233590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cw2m\" (UniqueName: \"kubernetes.io/projected/f8405899-6d85-4780-b49c-c403654373cb-kube-api-access-8cw2m\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.727820 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.727796    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.729889 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.729869    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:17.932752 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:17.932729    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:18.056157 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:18.056133    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"]
Jul 28 18:37:18.058446 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:37:18.058413    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8405899_6d85_4780_b49c_c403654373cb.slice/crio-6190305646bb1cb9896154832948ea2a84b3816c26d4056b02b0fc7095490c7b WatchSource:0}: Error finding container 6190305646bb1cb9896154832948ea2a84b3816c26d4056b02b0fc7095490c7b: Status 404 returned error can't find the container with id 6190305646bb1cb9896154832948ea2a84b3816c26d4056b02b0fc7095490c7b
Jul 28 18:37:18.089656 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:18.089635    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" event={"ID":"f8405899-6d85-4780-b49c-c403654373cb","Type":"ContainerStarted","Data":"6190305646bb1cb9896154832948ea2a84b3816c26d4056b02b0fc7095490c7b"}
Jul 28 18:37:19.094386 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:19.094353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" event={"ID":"f8405899-6d85-4780-b49c-c403654373cb","Type":"ContainerStarted","Data":"b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def"}
Jul 28 18:37:22.106161 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:22.106133    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg_f8405899-6d85-4780-b49c-c403654373cb/storage-initializer/0.log"
Jul 28 18:37:22.106579 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:22.106175    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8405899-6d85-4780-b49c-c403654373cb" containerID="b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def" exitCode=1
Jul 28 18:37:22.106579 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:22.106220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" event={"ID":"f8405899-6d85-4780-b49c-c403654373cb","Type":"ContainerDied","Data":"b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def"}
Jul 28 18:37:23.111635 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:23.111606    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg_f8405899-6d85-4780-b49c-c403654373cb/storage-initializer/0.log"
Jul 28 18:37:23.112116 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:23.111689    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" event={"ID":"f8405899-6d85-4780-b49c-c403654373cb","Type":"ContainerStarted","Data":"81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7"}
Jul 28 18:37:27.009707 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:27.009677    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"]
Jul 28 18:37:27.010170 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:27.009852    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer" containerID="cri-o://81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7" gracePeriod=30
Jul 28 18:37:28.539797 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.539775    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg_f8405899-6d85-4780-b49c-c403654373cb/storage-initializer/1.log"
Jul 28 18:37:28.540127 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.540113    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg_f8405899-6d85-4780-b49c-c403654373cb/storage-initializer/0.log"
Jul 28 18:37:28.542394 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.542380    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:28.597876 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.597854    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls\") pod \"f8405899-6d85-4780-b49c-c403654373cb\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") "
Jul 28 18:37:28.597968 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.597895    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8405899-6d85-4780-b49c-c403654373cb-kserve-provision-location\") pod \"f8405899-6d85-4780-b49c-c403654373cb\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") "
Jul 28 18:37:28.597968 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.597935    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cw2m\" (UniqueName: \"kubernetes.io/projected/f8405899-6d85-4780-b49c-c403654373cb-kube-api-access-8cw2m\") pod \"f8405899-6d85-4780-b49c-c403654373cb\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") "
Jul 28 18:37:28.598081 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.597984    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8405899-6d85-4780-b49c-c403654373cb-kube-rbac-proxy-sar-config\") pod \"f8405899-6d85-4780-b49c-c403654373cb\" (UID: \"f8405899-6d85-4780-b49c-c403654373cb\") "
Jul 28 18:37:28.598228 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.598204    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8405899-6d85-4780-b49c-c403654373cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8405899-6d85-4780-b49c-c403654373cb" (UID: "f8405899-6d85-4780-b49c-c403654373cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:28.598414 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.598383    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8405899-6d85-4780-b49c-c403654373cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8405899-6d85-4780-b49c-c403654373cb" (UID: "f8405899-6d85-4780-b49c-c403654373cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:28.599782 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.599754    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8405899-6d85-4780-b49c-c403654373cb" (UID: "f8405899-6d85-4780-b49c-c403654373cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:28.599856 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.599818    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8405899-6d85-4780-b49c-c403654373cb-kube-api-access-8cw2m" (OuterVolumeSpecName: "kube-api-access-8cw2m") pod "f8405899-6d85-4780-b49c-c403654373cb" (UID: "f8405899-6d85-4780-b49c-c403654373cb"). InnerVolumeSpecName "kube-api-access-8cw2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:28.698379 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.698330    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8405899-6d85-4780-b49c-c403654373cb-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:37:28.698379 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.698349    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8405899-6d85-4780-b49c-c403654373cb-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:37:28.698379 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.698358    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cw2m\" (UniqueName: \"kubernetes.io/projected/f8405899-6d85-4780-b49c-c403654373cb-kube-api-access-8cw2m\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:37:28.698379 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:28.698367    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8405899-6d85-4780-b49c-c403654373cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:37:29.131154 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131097    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg_f8405899-6d85-4780-b49c-c403654373cb/storage-initializer/1.log"
Jul 28 18:37:29.131544 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131528    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg_f8405899-6d85-4780-b49c-c403654373cb/storage-initializer/0.log"
Jul 28 18:37:29.131614 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131569    2571 generic.go:356] "Generic (PLEG): container finished" podID="f8405899-6d85-4780-b49c-c403654373cb" containerID="81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7" exitCode=1
Jul 28 18:37:29.131658 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131624    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"
Jul 28 18:37:29.131658 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" event={"ID":"f8405899-6d85-4780-b49c-c403654373cb","Type":"ContainerDied","Data":"81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7"}
Jul 28 18:37:29.131731 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131689    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg" event={"ID":"f8405899-6d85-4780-b49c-c403654373cb","Type":"ContainerDied","Data":"6190305646bb1cb9896154832948ea2a84b3816c26d4056b02b0fc7095490c7b"}
Jul 28 18:37:29.131731 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.131724    2571 scope.go:117] "RemoveContainer" containerID="81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7"
Jul 28 18:37:29.139412 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.139396    2571 scope.go:117] "RemoveContainer" containerID="b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def"
Jul 28 18:37:29.145974 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.145961    2571 scope.go:117] "RemoveContainer" containerID="81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7"
Jul 28 18:37:29.146210 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:37:29.146193    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7\": container with ID starting with 81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7 not found: ID does not exist" containerID="81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7"
Jul 28 18:37:29.146305 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.146215    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7"} err="failed to get container status \"81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7\": rpc error: code = NotFound desc = could not find container \"81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7\": container with ID starting with 81ef0f1870a3f0315d1eada86a8b9654bcacd5d9e7840d5b7001cf708c88e5b7 not found: ID does not exist"
Jul 28 18:37:29.146305 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.146230    2571 scope.go:117] "RemoveContainer" containerID="b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def"
Jul 28 18:37:29.146454 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:37:29.146439    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def\": container with ID starting with b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def not found: ID does not exist" containerID="b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def"
Jul 28 18:37:29.146492 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.146458    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def"} err="failed to get container status \"b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def\": rpc error: code = NotFound desc = could not find container \"b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def\": container with ID starting with b577c5c9ed3f80b71810dfe1043fae8bf1b841b3babe198fed708d237cc05def not found: ID does not exist"
Jul 28 18:37:29.167366 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.167340    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"]
Jul 28 18:37:29.171021 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.171002    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-7hqzg"]
Jul 28 18:37:29.707372 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:37:29.707346    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8405899-6d85-4780-b49c-c403654373cb" path="/var/lib/kubelet/pods/f8405899-6d85-4780-b49c-c403654373cb/volumes"
Jul 28 18:38:59.306032 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.305998    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"]
Jul 28 18:38:59.306449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.306223    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer"
Jul 28 18:38:59.306449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.306233    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer"
Jul 28 18:38:59.306449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.306265    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer"
Jul 28 18:38:59.306449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.306271    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer"
Jul 28 18:38:59.306449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.306312    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer"
Jul 28 18:38:59.306449 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.306321    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8405899-6d85-4780-b49c-c403654373cb" containerName="storage-initializer"
Jul 28 18:38:59.309151 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.309136    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.311995 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.311970    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:38:59.312134 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.311973    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:38:59.312134 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.312008    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 18:38:59.312134 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.311999    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:38:59.313127 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.313108    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:38:59.313230 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.313128    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:38:59.319083 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.319065    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"]
Jul 28 18:38:59.402377 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.402349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.402499 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.402388    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwj4g\" (UniqueName: \"kubernetes.io/projected/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-api-access-lwj4g\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.402499 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.402459    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c82260f-ebeb-4337-ba45-f3b4266d297c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.402499 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.402493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c82260f-ebeb-4337-ba45-f3b4266d297c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.503588 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.503570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c82260f-ebeb-4337-ba45-f3b4266d297c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.503702 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.503598    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c82260f-ebeb-4337-ba45-f3b4266d297c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.503702 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.503623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.503869 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.503772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwj4g\" (UniqueName: \"kubernetes.io/projected/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-api-access-lwj4g\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.504015 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.503995    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c82260f-ebeb-4337-ba45-f3b4266d297c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.504158 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.504142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.505970 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.505950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c82260f-ebeb-4337-ba45-f3b4266d297c-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.514627 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.514606    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwj4g\" (UniqueName: \"kubernetes.io/projected/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-api-access-lwj4g\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.619180 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.619127    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:38:59.738958 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:38:59.738936    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"]
Jul 28 18:38:59.741170 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:38:59.741142    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c82260f_ebeb_4337_ba45_f3b4266d297c.slice/crio-bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18 WatchSource:0}: Error finding container bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18: Status 404 returned error can't find the container with id bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18
Jul 28 18:39:00.417614 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:00.417580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" event={"ID":"4c82260f-ebeb-4337-ba45-f3b4266d297c","Type":"ContainerStarted","Data":"e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43"}
Jul 28 18:39:00.418040 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:00.417636    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" event={"ID":"4c82260f-ebeb-4337-ba45-f3b4266d297c","Type":"ContainerStarted","Data":"bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18"}
Jul 28 18:39:02.425357 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:02.425294    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/0.log"
Jul 28 18:39:02.425357 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:02.425339    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerID="e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43" exitCode=1
Jul 28 18:39:02.425683 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:02.425405    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" event={"ID":"4c82260f-ebeb-4337-ba45-f3b4266d297c","Type":"ContainerDied","Data":"e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43"}
Jul 28 18:39:03.430662 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:03.430637    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/0.log"
Jul 28 18:39:03.431074 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:03.430714    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" event={"ID":"4c82260f-ebeb-4337-ba45-f3b4266d297c","Type":"ContainerStarted","Data":"97a48214d8fb6a0b0c9312c1af0960ae6720872b2cc3cafe01f7b3caad9c29cd"}
Jul 28 18:39:05.438493 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.438468    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/1.log"
Jul 28 18:39:05.438825 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.438813    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/0.log"
Jul 28 18:39:05.438874 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.438844    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerID="97a48214d8fb6a0b0c9312c1af0960ae6720872b2cc3cafe01f7b3caad9c29cd" exitCode=1
Jul 28 18:39:05.438935 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.438915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" event={"ID":"4c82260f-ebeb-4337-ba45-f3b4266d297c","Type":"ContainerDied","Data":"97a48214d8fb6a0b0c9312c1af0960ae6720872b2cc3cafe01f7b3caad9c29cd"}
Jul 28 18:39:05.438993 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.438983    2571 scope.go:117] "RemoveContainer" containerID="e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43"
Jul 28 18:39:05.439325 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.439309    2571 scope.go:117] "RemoveContainer" containerID="e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43"
Jul 28 18:39:05.448485 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:39:05.448466    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_kserve-ci-e2e-test(4c82260f-ebeb-4337-ba45-f3b4266d297c)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c"
Jul 28 18:39:05.448553 ip-10-0-143-61 kubenswrapper[2571]: E0728 18:39:05.448533    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_kserve-ci-e2e-test_4c82260f-ebeb-4337-ba45-f3b4266d297c_0 in pod sandbox bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18 from index: no such id: 'e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43'" containerID="e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43"
Jul 28 18:39:05.448596 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:05.448562    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_kserve-ci-e2e-test_4c82260f-ebeb-4337-ba45-f3b4266d297c_0 in pod sandbox bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18 from index: no such id: 'e9c069428c8e5ed71e6e6b85d3c42dd7eaeb1a9d5a69f53aa475e90242087a43'"
Jul 28 18:39:06.443865 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:06.443844    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/1.log"
Jul 28 18:39:09.288415 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.288386    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"]
Jul 28 18:39:09.408394 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.408372    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/1.log"
Jul 28 18:39:09.412076 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.412061    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:39:09.455497 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.455471    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw_4c82260f-ebeb-4337-ba45-f3b4266d297c/storage-initializer/1.log"
Jul 28 18:39:09.455613 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.455550    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw" event={"ID":"4c82260f-ebeb-4337-ba45-f3b4266d297c","Type":"ContainerDied","Data":"bff73185fc1e16435ce6f477d754eab1173cb50ea99626dc16761a10bec7ff18"}
Jul 28 18:39:09.455613 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.455571    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"
Jul 28 18:39:09.455613 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.455608    2571 scope.go:117] "RemoveContainer" containerID="97a48214d8fb6a0b0c9312c1af0960ae6720872b2cc3cafe01f7b3caad9c29cd"
Jul 28 18:39:09.466947 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.466930    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c82260f-ebeb-4337-ba45-f3b4266d297c-proxy-tls\") pod \"4c82260f-ebeb-4337-ba45-f3b4266d297c\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") "
Jul 28 18:39:09.467019 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.466967    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-rbac-proxy-sar-config\") pod \"4c82260f-ebeb-4337-ba45-f3b4266d297c\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") "
Jul 28 18:39:09.467019 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.466992    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c82260f-ebeb-4337-ba45-f3b4266d297c-kserve-provision-location\") pod \"4c82260f-ebeb-4337-ba45-f3b4266d297c\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") "
Jul 28 18:39:09.467019 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.467008    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwj4g\" (UniqueName: \"kubernetes.io/projected/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-api-access-lwj4g\") pod \"4c82260f-ebeb-4337-ba45-f3b4266d297c\" (UID: \"4c82260f-ebeb-4337-ba45-f3b4266d297c\") "
Jul 28 18:39:09.467233 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.467208    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c82260f-ebeb-4337-ba45-f3b4266d297c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c82260f-ebeb-4337-ba45-f3b4266d297c" (UID: "4c82260f-ebeb-4337-ba45-f3b4266d297c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:09.467297 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.467241    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c82260f-ebeb-4337-ba45-f3b4266d297c" (UID: "4c82260f-ebeb-4337-ba45-f3b4266d297c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:09.468775 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.468750    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c82260f-ebeb-4337-ba45-f3b4266d297c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c82260f-ebeb-4337-ba45-f3b4266d297c" (UID: "4c82260f-ebeb-4337-ba45-f3b4266d297c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:09.468860 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.468830    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-api-access-lwj4g" (OuterVolumeSpecName: "kube-api-access-lwj4g") pod "4c82260f-ebeb-4337-ba45-f3b4266d297c" (UID: "4c82260f-ebeb-4337-ba45-f3b4266d297c"). InnerVolumeSpecName "kube-api-access-lwj4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:09.567430 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.567383    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c82260f-ebeb-4337-ba45-f3b4266d297c-proxy-tls\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:39:09.567430 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.567402    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:39:09.567430 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.567410    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c82260f-ebeb-4337-ba45-f3b4266d297c-kserve-provision-location\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:39:09.567430 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.567418    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwj4g\" (UniqueName: \"kubernetes.io/projected/4c82260f-ebeb-4337-ba45-f3b4266d297c-kube-api-access-lwj4g\") on node \"ip-10-0-143-61.ec2.internal\" DevicePath \"\""
Jul 28 18:39:09.797906 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.797885    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"]
Jul 28 18:39:09.803654 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:09.803633    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-wmmmw"]
Jul 28 18:39:11.706769 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:11.706738    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c" path="/var/lib/kubelet/pods/4c82260f-ebeb-4337-ba45-f3b4266d297c/volumes"
Jul 28 18:39:48.895553 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:48.895522    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5rqtb_e5b4bddb-be20-4289-bc3b-0b64b94a71b7/global-pull-secret-syncer/0.log"
Jul 28 18:39:49.047822 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:49.047790    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5mkh5_2882d280-bc58-4983-89f7-bfc3273fee8f/konnectivity-agent/0.log"
Jul 28 18:39:49.176515 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:49.176453    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-61.ec2.internal_a1804774383ef499fd25a45e880ee305/haproxy/0.log"
Jul 28 18:39:53.817652 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:53.817626    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slc24_03f51ace-0d90-439a-b426-9d3e9ce1b000/node-exporter/0.log"
Jul 28 18:39:53.839989 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:53.839961    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slc24_03f51ace-0d90-439a-b426-9d3e9ce1b000/kube-rbac-proxy/0.log"
Jul 28 18:39:53.865932 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:53.865912    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slc24_03f51ace-0d90-439a-b426-9d3e9ce1b000/init-textfile/0.log"
Jul 28 18:39:57.477056 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.477024    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"]
Jul 28 18:39:57.477455 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.477268    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerName="storage-initializer"
Jul 28 18:39:57.477455 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.477278    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerName="storage-initializer"
Jul 28 18:39:57.477455 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.477294    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerName="storage-initializer"
Jul 28 18:39:57.477455 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.477299    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerName="storage-initializer"
Jul 28 18:39:57.477455 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.477339    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c82260f-ebeb-4337-ba45-f3b4266d297c" containerName="storage-initializer"
Jul 28 18:39:57.480030 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.480011    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.482509 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.482490    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgbts\"/\"openshift-service-ca.crt\""
Jul 28 18:39:57.482633 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.482567    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kgbts\"/\"default-dockercfg-rnxxc\""
Jul 28 18:39:57.482690 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.482673    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgbts\"/\"kube-root-ca.crt\""
Jul 28 18:39:57.487515 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.487496    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"]
Jul 28 18:39:57.563439 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.563416    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-lib-modules\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.563524 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.563456    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-podres\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.563524 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.563476    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-sys\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.563598 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.563528    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6zs\" (UniqueName: \"kubernetes.io/projected/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-kube-api-access-qs6zs\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.563598 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.563550    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-proc\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664241 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-podres\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664354 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-sys\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664354 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6zs\" (UniqueName: \"kubernetes.io/projected/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-kube-api-access-qs6zs\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664354 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-proc\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664492 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-lib-modules\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664492 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664350    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-sys\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664492 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664395    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-podres\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664492 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-proc\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.664492 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.664476    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-lib-modules\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.672348 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.672325    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6zs\" (UniqueName: \"kubernetes.io/projected/7ed7c025-aa9a-4ad6-bde9-09a87b2320df-kube-api-access-qs6zs\") pod \"perf-node-gather-daemonset-x2vtz\" (UID: \"7ed7c025-aa9a-4ad6-bde9-09a87b2320df\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.789580 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.789536    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:57.909225 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:57.909197    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"]
Jul 28 18:39:57.912019 ip-10-0-143-61 kubenswrapper[2571]: W0728 18:39:57.911977    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7ed7c025_aa9a_4ad6_bde9_09a87b2320df.slice/crio-9ac40d059d9e6613285b275f5aa26859220641aa7d66e282b5764f8582eebd78 WatchSource:0}: Error finding container 9ac40d059d9e6613285b275f5aa26859220641aa7d66e282b5764f8582eebd78: Status 404 returned error can't find the container with id 9ac40d059d9e6613285b275f5aa26859220641aa7d66e282b5764f8582eebd78
Jul 28 18:39:58.257847 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.257828    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6k8qm_a5a94646-a349-4508-8f0d-b7511f8c1c44/dns/0.log"
Jul 28 18:39:58.280837 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.280819    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6k8qm_a5a94646-a349-4508-8f0d-b7511f8c1c44/kube-rbac-proxy/0.log"
Jul 28 18:39:58.407316 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.407295    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hll47_3eb4e5d0-96cd-4008-a7a0-d75e4df4493e/dns-node-resolver/0.log"
Jul 28 18:39:58.610425 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.610373    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz" event={"ID":"7ed7c025-aa9a-4ad6-bde9-09a87b2320df","Type":"ContainerStarted","Data":"e164c2e16ef82f521817b70b0159a62b31c698bed7d47dbebb1477df1e00f6f4"}
Jul 28 18:39:58.610425 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.610413    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz" event={"ID":"7ed7c025-aa9a-4ad6-bde9-09a87b2320df","Type":"ContainerStarted","Data":"9ac40d059d9e6613285b275f5aa26859220641aa7d66e282b5764f8582eebd78"}
Jul 28 18:39:58.610749 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.610502    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:39:58.625726 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.625691    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz" podStartSLOduration=1.625679582 podStartE2EDuration="1.625679582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:58.625111528 +0000 UTC m=+5383.385137147" watchObservedRunningTime="2026-07-28 18:39:58.625679582 +0000 UTC m=+5383.385705232"
Jul 28 18:39:58.978369 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:39:58.978344    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h5b8n_6553e69a-e7c5-4d46-b1d1-3ba4a443b68b/node-ca/0.log"
Jul 28 18:40:00.039031 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:00.039008    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jc2sx_b1a87c97-7a97-4e12-8caa-7f2d96d61ffc/serve-healthcheck-canary/0.log"
Jul 28 18:40:00.476255 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:00.476232    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cwvb_43b6fcec-f787-4ab0-aa8e-bb5506a6688b/kube-rbac-proxy/0.log"
Jul 28 18:40:00.524403 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:00.524386    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cwvb_43b6fcec-f787-4ab0-aa8e-bb5506a6688b/exporter/0.log"
Jul 28 18:40:00.555389 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:00.555369    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cwvb_43b6fcec-f787-4ab0-aa8e-bb5506a6688b/extractor/0.log"
Jul 28 18:40:02.991002 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:02.990973    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-jlqx5_52c23bf1-afb2-4063-a2e3-a7ea22b12218/manager/0.log"
Jul 28 18:40:03.012587 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:03.012562    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-rss2s_fd449bba-0f9a-435b-8cf6-db43b7f72141/server/0.log"
Jul 28 18:40:03.507873 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:03.507847    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-x69hq_b4ada816-f2f6-4d33-a58f-de33105a831d/seaweedfs-tls-serving/0.log"
Jul 28 18:40:04.622203 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:04.622176    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-x2vtz"
Jul 28 18:40:09.029329 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.029266    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4lvf4_da004608-b719-4ddd-b8fa-e55f4cb34081/kube-multus/0.log"
Jul 28 18:40:09.090694 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.090669    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:40:09.113320 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.113294    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/egress-router-binary-copy/0.log"
Jul 28 18:40:09.134907 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.134884    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/cni-plugins/0.log"
Jul 28 18:40:09.156753 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.156732    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/bond-cni-plugin/0.log"
Jul 28 18:40:09.178179 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.178156    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/routeoverride-cni/0.log"
Jul 28 18:40:09.199713 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.199693    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/whereabouts-cni-bincopy/0.log"
Jul 28 18:40:09.220540 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.220524    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hvthd_df7598ba-6577-4948-9df7-ba4528ee76b9/whereabouts-cni/0.log"
Jul 28 18:40:09.722893 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.722868    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z4x8n_0b662c0c-d267-40aa-8c94-748caf5b4735/network-metrics-daemon/0.log"
Jul 28 18:40:09.740936 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:09.740911    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z4x8n_0b662c0c-d267-40aa-8c94-748caf5b4735/kube-rbac-proxy/0.log"
Jul 28 18:40:10.878400 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:10.878362    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/ovn-controller/0.log"
Jul 28 18:40:10.894637 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:10.894594    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/ovn-acl-logging/0.log"
Jul 28 18:40:10.922194 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:10.922176    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/kube-rbac-proxy-node/0.log"
Jul 28 18:40:10.944375 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:10.944338    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:40:10.963770 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:10.963751    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/northd/0.log"
Jul 28 18:40:10.985444 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:10.985426    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/nbdb/0.log"
Jul 28 18:40:11.007704 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:11.007688    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/sbdb/0.log"
Jul 28 18:40:11.084286 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:11.084266    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wl2tm_70c2aa93-4685-4fe5-a1c6-eecf73c43442/ovnkube-controller/0.log"
Jul 28 18:40:12.055760 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:12.055729    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-22bpl_41dbd98a-cb15-4862-8ff6-b516258b3076/network-check-target-container/0.log"
Jul 28 18:40:13.014684 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:13.014661    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-96tjh_1e62a1f6-bb7f-4671-abb0-021a0aed58cf/iptables-alerter/0.log"
Jul 28 18:40:13.670400 ip-10-0-143-61 kubenswrapper[2571]: I0728 18:40:13.670367    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-w9m2f_a71ac77c-691c-4723-bc03-974cd9990fdb/tuned/0.log"