Jul 24 21:15:18.692954 ip-10-0-143-243 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 21:15:19.077348 ip-10-0-143-243 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:19.077348 ip-10-0-143-243 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 21:15:19.077348 ip-10-0-143-243 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:19.077348 ip-10-0-143-243 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 21:15:19.077348 ip-10-0-143-243 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:19.078324 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.078240    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 21:15:19.082405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082389    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:19.082405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082405    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082409    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082412    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082416    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082419    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082422    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082425    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082427    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082430    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082433    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082436    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082438    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082441    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082444    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082446    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082450    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082452    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082455    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082457    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:19.082472 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082461    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082465    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082468    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082471    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082474    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082476    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082479    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082482    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082485    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082487    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082490    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082493    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082495    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082498    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082501    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082503    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082506    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082508    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082511    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082513    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:19.082933 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082516    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082518    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082521    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082523    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082526    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082528    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082531    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082534    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082537    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082540    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082542    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082545    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082549    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082552    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082555    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082558    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082562    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082565    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082567    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:19.083409 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082570    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082573    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082576    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082579    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082581    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082584    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082586    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082589    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082591    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082594    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082596    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082599    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082601    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082604    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082606    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082609    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082613    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082617    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082620    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082622    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:19.083907 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082625    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082628    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082630    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082633    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082635    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082638    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082640    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082644    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082647    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082650    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082653    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082656    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082659    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082675    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082678    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082681    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082684    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082687    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082690    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:19.084393 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.082692    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083373    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083379    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083382    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083385    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083387    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083390    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083393    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083395    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083398    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083401    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083404    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083407    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083410    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083413    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083416    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083419    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083421    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083425    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083428    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:19.084917 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083430    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083433    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083435    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083438    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083440    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083443    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083445    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083448    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083450    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083453    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083455    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083457    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083460    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083462    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083465    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083467    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083470    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083472    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083475    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083477    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083480    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:19.085405 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083482    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083484    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083488    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083491    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083494    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083496    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083499    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083503    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083508    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083512    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083515    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083518    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083521    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083524    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083526    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083529    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083532    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083534    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:19.086082 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083537    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083541    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083545    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083547    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083550    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083552    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083556    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083559    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083562    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083566    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083569    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083571    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083574    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083577    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083579    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083581    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083584    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083588    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083591    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:19.086538 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083593    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083596    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083598    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083601    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083604    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083606    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083609    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083611    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083614    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083617    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083619    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083622    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083624    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083626    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083629    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083631    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083633    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083636    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083639    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083641    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:19.087024 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083643    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.083646    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084043    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084048    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084051    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084053    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084056    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084059    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084061    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084064    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084072    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084075    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084078    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084080    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084084    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084088    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084093    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084096    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084100    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:19.087509 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084102    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084105    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084108    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084110    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084113    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084115    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084118    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084120    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084122    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084125    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084127    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084130    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084133    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084135    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084137    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084140    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084142    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084145    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084147    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084150    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:19.087978 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084152    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084154    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084157    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084159    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084163    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084165    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084168    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084171    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084174    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084176    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084179    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084181    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084184    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084187    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084189    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084192    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084194    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084196    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084199    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084201    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:19.088461 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084203    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084206    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084208    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084211    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084213    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084215    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084217    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084220    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084222    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084225    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084227    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084230    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084232    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084235    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084237    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084240    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084242    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084245    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084248    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084251    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:19.089013 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084255    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084258    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084260    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084263    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084265    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084267    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084270    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084272    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084275    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084279    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084283    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084286    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084288    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084291    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084294    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084297    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084299    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084302    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084304    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:19.089493 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084307    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084309    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.084312    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084383    2585 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084389    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084395    2585 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084400    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084404    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084409    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084416    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084422    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084428    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084434    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084438    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084441    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084444    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084447    2585 flags.go:64] FLAG: --cgroup-root=""
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084450    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084452    2585 flags.go:64] FLAG: --client-ca-file=""
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084455    2585 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084458    2585 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084462    2585 flags.go:64] FLAG: --cluster-domain=""
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084464    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084467    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 21:15:19.089960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084470    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084473    2585 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084477    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084480    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084483    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084486    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084489    2585 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084492    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084494    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084497    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084500    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084504    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084507    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084510    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084512    2585 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084531    2585 flags.go:64] FLAG: --enable-server="true"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084536    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084541    2585 flags.go:64] FLAG: --event-burst="100"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084545    2585 flags.go:64] FLAG: --event-qps="50"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084548    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084552    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084555    2585 flags.go:64] FLAG: --eviction-hard=""
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084558    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084561    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084564    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 21:15:19.090548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084568    2585 flags.go:64] FLAG: --eviction-soft=""
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084571    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084573    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084576    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084579    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084582    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084585    2585 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084588    2585 flags.go:64] FLAG: --feature-gates=""
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084591    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084595    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084597    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084600    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084603    2585 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084606    2585 flags.go:64] FLAG: --help="false"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084609    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084612    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084615    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084618    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084621    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084625    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084628    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084630    2585 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084633    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 21:15:19.091220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084636    2585 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084639    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084642    2585 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084645    2585 flags.go:64] FLAG: --kube-reserved=""
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084648    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084651    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084655    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084658    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084673    2585 flags.go:64] FLAG: --lock-file=""
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084676    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084679    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084682    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084692    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084695    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084698    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084701    2585 flags.go:64] FLAG: --logging-format="text"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084703    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084707    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084709    2585 flags.go:64] FLAG: --manifest-url=""
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084712    2585 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084716    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084719    2585 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084723    2585 flags.go:64] FLAG: --max-pods="110"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084726    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084729    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 21:15:19.091872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084732    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084735    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084737    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084740    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084743    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084750    2585 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084753    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084756    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084759    2585 flags.go:64] FLAG: --pod-cidr=""
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084762    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084767    2585 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084770    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084773    2585 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084776    2585 flags.go:64] FLAG: --port="10250"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084779    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084783    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b3194b4bc25f51bc"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084786    2585 flags.go:64] FLAG: --qos-reserved=""
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084788    2585 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084791    2585 flags.go:64] FLAG: --register-node="true"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084794    2585 flags.go:64] FLAG: --register-with-taints=""
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084797    2585 flags.go:64] FLAG: --registry-burst="10"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084800    2585 flags.go:64] FLAG: --registry-qps="5"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084803    2585 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084805    2585 flags.go:64] FLAG: --reserved-memory=""
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084809    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 21:15:19.092474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084814    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084816    2585 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084819    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084822    2585 flags.go:64] FLAG: --runonce="false"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084824    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084827    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084830    2585 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084835    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084838    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084841    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084844    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084847    2585 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084850    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084852    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084855    2585 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084858    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084861    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084864    2585 flags.go:64] FLAG: --system-cgroups=""
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084866    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084871    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084874    2585 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084877    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084881    2585 flags.go:64] FLAG: --tls-min-version=""
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084884    2585 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084886    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 21:15:19.093080 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084889    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084892    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084895    2585 flags.go:64] FLAG: --v="2"
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084898    2585 flags.go:64] FLAG: --version="false"
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084902    2585 flags.go:64] FLAG: --vmodule=""
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084905    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.084908    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085040    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085044    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085047    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085050    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085053    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085056    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085058    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085063    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085065    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085068    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085070    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085073    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085075    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085078    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:19.093686 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085080    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085082    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085085    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085087    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085090    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085092    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085094    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085097    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085100    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085103    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085106    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085108    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085111    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085113    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085115    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085118    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085120    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085123    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085126    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:19.094206 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085130    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085134    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085137    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085139    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085142    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085144    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085148    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085151    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085153    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085156    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085158    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085160    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085163    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085166    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085170    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085173    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085176    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085179    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085181    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085183    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:19.094736 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085186    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085188    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085191    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085194    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085196    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085198    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085201    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085203    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085206    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085208    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085211    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085216    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085219    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085222    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085225    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085227    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085230    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085232    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085236    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085239    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:19.095244 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085242    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085244    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085246    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085249    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085251    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085254    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085256    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085259    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085262    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085265    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085267    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085270    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085272    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085275    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085278    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085281    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085283    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085286    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085288    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:19.095744 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085290    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085293    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085295    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085297    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085301    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085304    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.085306    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.085898    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.092467    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.092483    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.093719    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.093727    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093797    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093803    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093806    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:19.096198 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093809    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093813    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093815    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093818    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093821    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093824    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093826    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093829    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093831    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093833    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093836    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093838    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093842    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093846    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093849    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093851    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093854    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093857    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093859    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093861    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:19.096564 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093864    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093866    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093869    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093871    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093874    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093876    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093880    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093883    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093885    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093888    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093891    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093893    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093896    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093898    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093900    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093903    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093905    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093908    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093910    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093912    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:19.097054 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093915    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093918    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093920    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093923    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093925    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093927    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093930    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093932    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093934    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093937    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093939    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093942    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093944    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093946    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093948    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093951    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093954    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093956    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093959    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093963    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:19.097523 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093965    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093969    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093972    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093975    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093978    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093989    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093993    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093997    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.093999    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094002    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094004    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094007    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094010    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094012    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094015    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094017    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094019    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094022    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094024    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:19.098008 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094027    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094029    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094032    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094034    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094037    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094039    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094042    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094044    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094046    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094049    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094052    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094055    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094057    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094061    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094063    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094066    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:19.098496 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094068    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.094073    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094181    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094186    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094189    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094191    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094194    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094197    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094199    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094202    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094205    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094207    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094210    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094213    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:19.098976 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094215    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094219    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094221    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094224    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094226    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094229    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094231    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094233    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094236    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094238    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094240    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094243    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094245    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094248    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094251    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094253    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094256    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094258    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094262    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:19.099317 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094265    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094269    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094273    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094276    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094278    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094281    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094283    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094286    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094288    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094290    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094293    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094295    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094297    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094300    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094302    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094305    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094307    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094309    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094311    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:19.099792 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094315    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094318    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094321    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094324    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094326    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094329    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094331    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094334    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094336    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094339    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094341    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094344    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094346    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094349    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094352    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094354    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094356    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094359    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094361    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094364    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:19.100259 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094366    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094368    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094371    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094373    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094375    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094378    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094380    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094382    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094385    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094387    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094390    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094392    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094394    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094397    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094399    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094402    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094404    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094407    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094409    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094412    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:19.100748 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094415    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094417    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094420    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094422    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094425    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094427    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094430    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094432    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:19.094434    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.094439    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.094543    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.096415    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.099073    2585 server.go:1019] "Starting client certificate rotation"
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.099179    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:15:19.101236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.099209    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:15:19.102208 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.102195    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:15:19.105711 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.105695    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:15:19.119705 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.119687    2585 log.go:25] "Validated CRI v1 runtime API"
Jul 24 21:15:19.121623 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.121605    2585 log.go:25] "Validated CRI v1 image API"
Jul 24 21:15:19.123479 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.123454    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 21:15:19.125777 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.125747    2585 fs.go:135] Filesystem UUIDs: map[19c6a19a-6c38-41ff-943d-80322ef7dea3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 81dddb1b-66a2-4d67-85bc-1fe12ada4d42:/dev/nvme0n1p4]
Jul 24 21:15:19.125861 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.125775    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 21:15:19.130940 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.130841    2585 manager.go:216] Machine: {Timestamp:2026-07-24 21:15:19.130359798 +0000 UTC m=+0.323537746 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097809 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2302a6311d1c353df955aee2a696a0 SystemUUID:ec2302a6-311d-1c35-3df9-55aee2a696a0 BootID:ddbf0d6a-5eb6-4db5-af3f-bdb0d27265a5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:98:59:7d:92:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:98:59:7d:92:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:e5:d1:29:e4:e0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 21:15:19.130940 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.130936    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 21:15:19.131039 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.131003    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 21:15:19.134369 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.134342    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 21:15:19.134496 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.134374    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-243.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 21:15:19.135776 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.135766    2585 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 21:15:19.135815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.135778    2585 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 21:15:19.135815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.135786    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:15:19.136310 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.136300    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:15:19.137727 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.137716    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 21:15:19.140352 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140341    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:15:19.140479 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140470    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 21:15:19.140512 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140501    2585 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 21:15:19.140540 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140524    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 21:15:19.140568 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140543    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 21:15:19.140568 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140555    2585 kubelet.go:398] "Adding apiserver pod source"
Jul 24 21:15:19.140568 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140567    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 21:15:19.140979 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140964    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:15:19.141028 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.140993    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:15:19.141519 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.141503    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:15:19.143073 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.143038    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 21:15:19.144517 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.144504    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 21:15:19.144571 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.144542    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 21:15:19.145153 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.145143    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 21:15:19.145190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.145158    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 21:15:19.145190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.145164    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 21:15:19.145190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.145170    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 21:15:19.145190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.145177    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 21:15:19.145190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.145183    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 21:15:19.146311 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.146300    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 21:15:19.146343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.146311    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 21:15:19.146343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.146318    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 21:15:19.146343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.146324    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 21:15:19.146343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.146338    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 21:15:19.146343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.146345    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 21:15:19.147190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.147179    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 21:15:19.147190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.147190    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 21:15:19.149751 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.149730    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-243.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 21:15:19.149835 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.149759    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 21:15:19.149875 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.149849    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 21:15:19.150791 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.150778    2585 server.go:1268] "Started kubelet"
Jul 24 21:15:19.150883 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.150852    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 21:15:19.150934 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.150870    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 21:15:19.150972 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.150947    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 21:15:19.151485 ip-10-0-143-243 systemd[1]: Started Kubernetes Kubelet.
Jul 24 21:15:19.155862 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.155835    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 21:15:19.157845 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.157818    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 21:15:19.158112 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.158098    2585 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 21:15:19.164098 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.164073    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 21:15:19.164098 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.164081    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 21:15:19.164308 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.164220    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165200    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165217    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165224    2585 factory.go:153] Registering CRI-O factory
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165242    2585 factory.go:223] Registration of the crio container factory successfully
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165291    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165293    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 21:15:19.165309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165313    2585 factory.go:55] Registering systemd factory
Jul 24 21:15:19.165585 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165332    2585 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 21:15:19.165585 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165342    2585 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 21:15:19.165585 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165334    2585 factory.go:223] Registration of the systemd container factory successfully
Jul 24 21:15:19.165585 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165372    2585 factory.go:103] Registering Raw factory
Jul 24 21:15:19.165585 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165387    2585 manager.go:1196] Started watching for new ooms in manager
Jul 24 21:15:19.165816 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.165803    2585 manager.go:349] Starting recovery of all containers
Jul 24 21:15:19.169000 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.168978    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ssd64"
Jul 24 21:15:19.169558 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.169533    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 21:15:19.169779 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.169733    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 21:15:19.170645 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.169652    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-243.ec2.internal.18c5564db9b77b4f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-243.ec2.internal,UID:ip-10-0-143-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-243.ec2.internal,},FirstTimestamp:2026-07-24 21:15:19.150549839 +0000 UTC m=+0.343727766,LastTimestamp:2026-07-24 21:15:19.150549839 +0000 UTC m=+0.343727766,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-243.ec2.internal,}"
Jul 24 21:15:19.176024 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.176007    2585 manager.go:354] Recovery completed
Jul 24 21:15:19.176526 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.176504    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ssd64"
Jul 24 21:15:19.180364 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.180347    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.182061 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182045    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.182151 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182077    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.182151 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182091    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.182577 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182565    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 21:15:19.182577 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182577    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 21:15:19.182652 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182591    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:15:19.182652 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.182628    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.183310 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.183296    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.183358 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.183321    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.183358 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.183331    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.183570 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.183558    2585 policy_none.go:49] "None policy: Start"
Jul 24 21:15:19.183601 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.183575    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 21:15:19.183601 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.183585    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 21:15:19.183991 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.183910    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-243.ec2.internal.18c5564dbb984f6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-243.ec2.internal,UID:ip-10-0-143-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-243.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-243.ec2.internal,},FirstTimestamp:2026-07-24 21:15:19.182061418 +0000 UTC m=+0.375239353,LastTimestamp:2026-07-24 21:15:19.182061418 +0000 UTC m=+0.375239353,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-243.ec2.internal,}"
Jul 24 21:15:19.184738 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.184726    2585 policy_none.go:47] "Start"
Jul 24 21:15:19.225212 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225190    2585 manager.go:337] "Starting Device Plugin manager"
Jul 24 21:15:19.225312 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.225240    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 21:15:19.225312 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225255    2585 server.go:87] "Starting device plugin registration server"
Jul 24 21:15:19.225500 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225486    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 21:15:19.225557 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225502    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 21:15:19.225608 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225558    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 21:15:19.225652 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225628    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 21:15:19.225652 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.225637    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 21:15:19.226266 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.226247    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 21:15:19.226342 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.226291    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.319938 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.319798    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 21:15:19.321250 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.321234    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 21:15:19.321317 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.321283    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 21:15:19.321317 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.321308    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 21:15:19.321408 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.321348    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 21:15:19.323343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.323328    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:19.326345 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.326331    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.327104 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.327080    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.327104 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.327101    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.327230 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.327111    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.327230 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.327132    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.334951 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.334912    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.334951 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.334934    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-243.ec2.internal\": node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.354860 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.354843    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.422092 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.422066    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"]
Jul 24 21:15:19.422170 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.422145    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.423614 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.423601    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.423681 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.423629    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.423681 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.423639    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.424838 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.424826    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.424966 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.424949    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.425014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.424982    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.425526 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.425512    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.425597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.425539    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.425597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.425553    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.425597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.425562    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.425597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.425582    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.425597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.425595    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.426855 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.426838    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.426942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.426861    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:19.427523 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.427500    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:19.427597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.427527    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:19.427597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.427538    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:19.450196 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.450176    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-243.ec2.internal\" not found" node="ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.454636 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.454623    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-243.ec2.internal\" not found" node="ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.455627 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.455612    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.466985 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.466968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1bea1286bfe6de533a5546f72ba9243-config\") pod \"kube-apiserver-proxy-ip-10-0-143-243.ec2.internal\" (UID: \"c1bea1286bfe6de533a5546f72ba9243\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.467078 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.466991    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b1dd8d37b48cd7553e4d70ea9d781b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal\" (UID: \"1b1dd8d37b48cd7553e4d70ea9d781b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.467078 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.467008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1dd8d37b48cd7553e4d70ea9d781b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal\" (UID: \"1b1dd8d37b48cd7553e4d70ea9d781b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.556202 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.556178    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.567508 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.567488    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1bea1286bfe6de533a5546f72ba9243-config\") pod \"kube-apiserver-proxy-ip-10-0-143-243.ec2.internal\" (UID: \"c1bea1286bfe6de533a5546f72ba9243\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.567563 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.567511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b1dd8d37b48cd7553e4d70ea9d781b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal\" (UID: \"1b1dd8d37b48cd7553e4d70ea9d781b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.567563 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.567529    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1dd8d37b48cd7553e4d70ea9d781b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal\" (UID: \"1b1dd8d37b48cd7553e4d70ea9d781b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.567625 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.567567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1dd8d37b48cd7553e4d70ea9d781b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal\" (UID: \"1b1dd8d37b48cd7553e4d70ea9d781b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.567625 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.567596    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b1dd8d37b48cd7553e4d70ea9d781b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal\" (UID: \"1b1dd8d37b48cd7553e4d70ea9d781b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.567625 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.567596    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c1bea1286bfe6de533a5546f72ba9243-config\") pod \"kube-apiserver-proxy-ip-10-0-143-243.ec2.internal\" (UID: \"c1bea1286bfe6de533a5546f72ba9243\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.656976 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.656911    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.752521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.752490    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.757641 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:19.757450    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:19.757641 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.757460    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.857716 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.857692    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:19.958325 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:19.958262    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.033976 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.033954    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:20.058989 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.058969    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.097523 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.097503    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 21:15:20.097897 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.097625    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 21:15:20.097897 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.097658    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 21:15:20.120488 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.120469    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:20.159096 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.159071    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.180343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.180310    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 21:10:19 +0000 UTC" deadline="2028-05-08 01:17:42.543875002 +0000 UTC"
Jul 24 21:15:20.180343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.180340    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15676h2m22.363539711s"
Jul 24 21:15:20.259630 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.259606    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.339720 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:20.339694    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bea1286bfe6de533a5546f72ba9243.slice/crio-adad3efa471e1da2dcbdc90552b34244f4e99128302ca82544d34625765cd555 WatchSource:0}: Error finding container adad3efa471e1da2dcbdc90552b34244f4e99128302ca82544d34625765cd555: Status 404 returned error can't find the container with id adad3efa471e1da2dcbdc90552b34244f4e99128302ca82544d34625765cd555
Jul 24 21:15:20.339904 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:20.339886    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1dd8d37b48cd7553e4d70ea9d781b1.slice/crio-1ddd40bcf30f749bbd7abbd90601b8d156cd92e9b349f5af19e088531cf38024 WatchSource:0}: Error finding container 1ddd40bcf30f749bbd7abbd90601b8d156cd92e9b349f5af19e088531cf38024: Status 404 returned error can't find the container with id 1ddd40bcf30f749bbd7abbd90601b8d156cd92e9b349f5af19e088531cf38024
Jul 24 21:15:20.344867 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.344851    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:15:20.359800 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.359782    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.462237 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.462182    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.565187 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.565150    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.665387 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:20.665344    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-243.ec2.internal\" not found"
Jul 24 21:15:20.679297 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.679273    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:20.765063 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.764990    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:20.778441 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.778420    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:15:20.779668 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.779651    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal"
Jul 24 21:15:20.788697 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:20.788662    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:15:21.141704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.141611    2585 apiserver.go:52] "Watching apiserver"
Jul 24 21:15:21.156009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.155578    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 21:15:21.156583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.156552    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal","openshift-cluster-node-tuning-operator/tuned-l7w94","openshift-image-registry/node-ca-tq265","openshift-multus/multus-additional-cni-plugins-8plsn","openshift-network-diagnostics/network-check-target-n6hjf","openshift-network-operator/iptables-alerter-v7hcf","openshift-ovn-kubernetes/ovnkube-node-6nm8n","kube-system/global-pull-secret-syncer-7kk4m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr","openshift-dns/node-resolver-f5bml","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal","openshift-multus/multus-gxvfx","openshift-multus/network-metrics-daemon-5sl4p","kube-system/konnectivity-agent-2hw84"]
Jul 24 21:15:21.159638 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.159617    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.162208 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.162182    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.162693 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.162658    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.163202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.163184    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mqv6j\""
Jul 24 21:15:21.163202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.163194    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.165058    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bcjlp\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.165290    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.165608    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.165837    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.166007    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.166388    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.166532    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.167018 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.166678    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 21:15:21.168356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.168065    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.170905 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.170883    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.171022 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.170957    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:21.171092 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.171075    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r7jjw\""
Jul 24 21:15:21.171229 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.171193    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 21:15:21.171362 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.171292    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 21:15:21.173853 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.173835    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:21.173951 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.173893    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:21.176116 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176092    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.176506 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176481    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-systemd-units\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.176613 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176523    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-run-netns\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.176613 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-etc-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.176613 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176559    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.176613 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176581    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.176613 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176609    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176634    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovnkube-script-lib\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-slash\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-systemd\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176804    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-ovn\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176848    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-log-socket\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176877    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-cni-bin\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176902    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176939    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-env-overrides\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.176982    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-var-lib-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhmg\" (UniqueName: \"kubernetes.io/projected/980a987a-0aa5-4f5f-8c93-ab877c736e8e-kube-api-access-8hhmg\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177049    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5452a9bc-1899-4ec2-ae0b-78039a6722e8-konnectivity-ca\") pod \"konnectivity-agent-2hw84\" (UID: \"5452a9bc-1899-4ec2-ae0b-78039a6722e8\") " pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177075    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/392c898d-7923-488d-b0c1-1da116ad9ec0-hosts-file\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54wv\" (UniqueName: \"kubernetes.io/projected/392c898d-7923-488d-b0c1-1da116ad9ec0-kube-api-access-f54wv\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177137    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-kubelet\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177232    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-node-log\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177283    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-cni-netd\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177307    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovnkube-config\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5452a9bc-1899-4ec2-ae0b-78039a6722e8-agent-certs\") pod \"konnectivity-agent-2hw84\" (UID: \"5452a9bc-1899-4ec2-ae0b-78039a6722e8\") " pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.177482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.177348    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392c898d-7923-488d-b0c1-1da116ad9ec0-tmp-dir\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.178929 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.178909    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.181116 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.181093    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.182419 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.182396    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkph7\""
Jul 24 21:15:21.182618 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.182599    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.182618 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.182624    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.182777 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.182659    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 21:15:21.182777 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.182605    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.183152 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.183133    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 21:15:21.183363 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.183350    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.183423 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.183379    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 21:15:21.183617 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.183597    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h6fx9\""
Jul 24 21:15:21.184973 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.184953    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 21:15:21.185239 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.185225    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.185318 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.185286    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 21:15:21.185399 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.185386    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xhm4w\""
Jul 24 21:15:21.185461 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.185431    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.185612 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.185593    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:15:21.186477 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.186457    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.188956 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.188931    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.189054 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.188962    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.189409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.189390    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.189520 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.189398    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7shdc\""
Jul 24 21:15:21.191700 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.191453    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.191700 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.191522    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:21.191880 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.191861    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8slhw\""
Jul 24 21:15:21.191946 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.191918    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 21:15:21.194020 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.193997    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.197336 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.197011    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 21:15:21.198782 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.198412    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c8kbp\""
Jul 24 21:15:21.198782 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.198653    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 21:15:21.198986 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.198961    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 21:15:21.208379 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.208353    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m4fl7"
Jul 24 21:15:21.215489 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.215144    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m4fl7"
Jul 24 21:15:21.267128 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.267103    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 21:15:21.278241 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278209    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/392c898d-7923-488d-b0c1-1da116ad9ec0-hosts-file\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.278351 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54wv\" (UniqueName: \"kubernetes.io/projected/392c898d-7923-488d-b0c1-1da116ad9ec0-kube-api-access-f54wv\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.278351 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278277    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-slash\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.278351 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278307    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c187ee75-0aff-41ff-be95-a05e6ca2cff3-iptables-alerter-script\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.278351 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278334    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-modprobe-d\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278358    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysctl-d\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278386    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdkh\" (UniqueName: \"kubernetes.io/projected/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-kube-api-access-9xdkh\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29jh\" (UniqueName: \"kubernetes.io/projected/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-kube-api-access-r29jh\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278437    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-socket-dir-parent\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-kubelet\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278496    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-run\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-var-lib-kubelet\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278556    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-tuned\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278583    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278610    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-system-cni-dir\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysctl-conf\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278698    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-lib-modules\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278724    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5452a9bc-1899-4ec2-ae0b-78039a6722e8-konnectivity-ca\") pod \"konnectivity-agent-2hw84\" (UID: \"5452a9bc-1899-4ec2-ae0b-78039a6722e8\") " pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278768    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-ovn\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278794    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-log-socket\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278820    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-env-overrides\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-os-release\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278875    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278900    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cnibin\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278936    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-host\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.278995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.278983    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-device-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-cni-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279033    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-cnibin\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279062    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-hostroot\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279097    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-cni-netd\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279123    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovnkube-config\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-sys\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279206    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-tmp\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279255    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-sys-fs\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-multus-certs\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-systemd-units\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279334    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-run-netns\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279361    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-k8s-cni-cncf-io\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279419    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-cni-bin\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.279622 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279459    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-conf-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279484    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kjw\" (UniqueName: \"kubernetes.io/projected/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-kube-api-access-q4kjw\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sgxj\" (UniqueName: \"kubernetes.io/projected/df23e468-1389-4fbc-9e50-06d5d97d3e1c-kube-api-access-9sgxj\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-systemd\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279595    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c187ee75-0aff-41ff-be95-a05e6ca2cff3-host-slash\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279655    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cni-binary-copy\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-system-cni-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279707    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df23e468-1389-4fbc-9e50-06d5d97d3e1c-host\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279734    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-var-lib-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279782    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69m52\" (UniqueName: \"kubernetes.io/projected/c187ee75-0aff-41ff-be95-a05e6ca2cff3-kube-api-access-69m52\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279817    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-registration-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279842    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-os-release\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279867    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-daemon-config\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-node-log\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5452a9bc-1899-4ec2-ae0b-78039a6722e8-agent-certs\") pod \"konnectivity-agent-2hw84\" (UID: \"5452a9bc-1899-4ec2-ae0b-78039a6722e8\") " pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysconfig\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.280521 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.279985    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-netns\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280011    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-systemd\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280038    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-etc-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovnkube-script-lib\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9gz\" (UniqueName: \"kubernetes.io/projected/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-kube-api-access-md9gz\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280122    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280151    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-cni-binary-copy\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280183    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-cni-multus\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-kubelet\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280242    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-cni-bin\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280270    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qlp\" (UniqueName: \"kubernetes.io/projected/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-kube-api-access-w5qlp\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df23e468-1389-4fbc-9e50-06d5d97d3e1c-serviceca\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280355    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhmg\" (UniqueName: \"kubernetes.io/projected/980a987a-0aa5-4f5f-8c93-ab877c736e8e-kube-api-access-8hhmg\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280383    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-etc-kubernetes\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280411    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-kubernetes\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.281326 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280438    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392c898d-7923-488d-b0c1-1da116ad9ec0-tmp-dir\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280497    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280523    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-socket-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280551    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-kubelet-config\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280581    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-dbus\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.280691    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/392c898d-7923-488d-b0c1-1da116ad9ec0-hosts-file\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281075    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281176    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-run-netns\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-systemd\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281258    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-var-lib-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281347    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-node-log\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281399    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-cni-bin\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281472    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-slash\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281524    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-kubelet\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281571    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281621    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-cni-netd\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.282202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.281865    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-env-overrides\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282088    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/392c898d-7923-488d-b0c1-1da116ad9ec0-tmp-dir\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282165    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282416    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5452a9bc-1899-4ec2-ae0b-78039a6722e8-konnectivity-ca\") pod \"konnectivity-agent-2hw84\" (UID: \"5452a9bc-1899-4ec2-ae0b-78039a6722e8\") " pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282471    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-run-ovn\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282506    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-log-socket\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-etc-openvswitch\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282613    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovnkube-config\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/980a987a-0aa5-4f5f-8c93-ab877c736e8e-systemd-units\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.283120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.282996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovnkube-script-lib\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.285697 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.285669    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5452a9bc-1899-4ec2-ae0b-78039a6722e8-agent-certs\") pod \"konnectivity-agent-2hw84\" (UID: \"5452a9bc-1899-4ec2-ae0b-78039a6722e8\") " pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.285958 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.285939    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/980a987a-0aa5-4f5f-8c93-ab877c736e8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.290446 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.290404    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54wv\" (UniqueName: \"kubernetes.io/projected/392c898d-7923-488d-b0c1-1da116ad9ec0-kube-api-access-f54wv\") pod \"node-resolver-f5bml\" (UID: \"392c898d-7923-488d-b0c1-1da116ad9ec0\") " pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.291687 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.291659    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhmg\" (UniqueName: \"kubernetes.io/projected/980a987a-0aa5-4f5f-8c93-ab877c736e8e-kube-api-access-8hhmg\") pod \"ovnkube-node-6nm8n\" (UID: \"980a987a-0aa5-4f5f-8c93-ab877c736e8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.326739 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.326668    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal" event={"ID":"1b1dd8d37b48cd7553e4d70ea9d781b1","Type":"ContainerStarted","Data":"1ddd40bcf30f749bbd7abbd90601b8d156cd92e9b349f5af19e088531cf38024"}
Jul 24 21:15:21.328333 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.328301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal" event={"ID":"c1bea1286bfe6de533a5546f72ba9243","Type":"ContainerStarted","Data":"adad3efa471e1da2dcbdc90552b34244f4e99128302ca82544d34625765cd555"}
Jul 24 21:15:21.381581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381545    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-run\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.381731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381597    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-var-lib-kubelet\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.381731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-tuned\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.381731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-system-cni-dir\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.381731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381669    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysctl-conf\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.381731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381712    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381735    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-lib-modules\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381764    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-os-release\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381771    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-run\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381789    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381814    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cnibin\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381840    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-host\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381888    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-device-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381913    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-cni-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381937    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-cnibin\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381959    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-hostroot\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.381988    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382007    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-os-release\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382007 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382013    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-sys\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-tmp\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382059    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382077    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-sys-fs\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382077    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-sys\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-multus-certs\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-k8s-cni-cncf-io\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-cni-bin\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382146    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-conf-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382161    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kjw\" (UniqueName: \"kubernetes.io/projected/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-kube-api-access-q4kjw\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.382176    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.382258    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:15:21.882218687 +0000 UTC m=+3.075396606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382311    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cnibin\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382354    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-var-lib-kubelet\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382532    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-host\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382587    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-device-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.382704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sgxj\" (UniqueName: \"kubernetes.io/projected/df23e468-1389-4fbc-9e50-06d5d97d3e1c-kube-api-access-9sgxj\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382711    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c187ee75-0aff-41ff-be95-a05e6ca2cff3-host-slash\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382725    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-cni-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382746    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cni-binary-copy\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382775    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-system-cni-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df23e468-1389-4fbc-9e50-06d5d97d3e1c-host\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382835    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69m52\" (UniqueName: \"kubernetes.io/projected/c187ee75-0aff-41ff-be95-a05e6ca2cff3-kube-api-access-69m52\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.382862    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-registration-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-system-cni-dir\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383024    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-os-release\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383067    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df23e468-1389-4fbc-9e50-06d5d97d3e1c-host\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-daemon-config\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysconfig\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383115    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-netns\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-cnibin\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-hostroot\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383201    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-netns\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.383581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383217    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-systemd\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383234    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-os-release\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383251    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9gz\" (UniqueName: \"kubernetes.io/projected/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-kube-api-access-md9gz\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383284    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysconfig\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383297    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-cni-binary-copy\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383296    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-registration-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-cni-multus\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383339    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysctl-conf\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-kubelet\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383393    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.383416    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qlp\" (UniqueName: \"kubernetes.io/projected/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-kube-api-access-w5qlp\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383427    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-cni-binary-copy\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383451    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df23e468-1389-4fbc-9e50-06d5d97d3e1c-serviceca\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.383470    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:15:21.883452522 +0000 UTC m=+3.076630454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383498    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-system-cni-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.384710 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383555    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-k8s-cni-cncf-io\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383579    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-lib-modules\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-cni-bin\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383664    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-conf-dir\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383858    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-cni-binary-copy\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383929    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c187ee75-0aff-41ff-be95-a05e6ca2cff3-host-slash\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383938    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df23e468-1389-4fbc-9e50-06d5d97d3e1c-serviceca\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383991    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-etc-kubernetes\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.383995    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-daemon-config\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384001    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-cni-multus\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384040    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-etc-kubernetes\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-kubernetes\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384092    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-systemd\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384097    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-socket-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-kubelet-config\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384177    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-var-lib-kubelet\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384201    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-dbus\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.385583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384241    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c187ee75-0aff-41ff-be95-a05e6ca2cff3-iptables-alerter-script\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-modprobe-d\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384277    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-kubernetes\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysctl-d\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdkh\" (UniqueName: \"kubernetes.io/projected/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-kube-api-access-9xdkh\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384342    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-dbus\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384146    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384359    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29jh\" (UniqueName: \"kubernetes.io/projected/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-kube-api-access-r29jh\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384402    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-socket-dir-parent\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384465    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-socket-dir\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-multus-socket-dir-parent\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-kubelet-config\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384630    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-sysctl-d\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384668    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-sys-fs\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384713    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384731    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-host-run-multus-certs\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.386559 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.384827    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-modprobe-d\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.387491 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.385109    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-etc-tuned\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.387491 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.385601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c187ee75-0aff-41ff-be95-a05e6ca2cff3-iptables-alerter-script\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.390520 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.390500    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-tmp\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.394294 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.394242    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:21.394294 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.394270    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:21.394294 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.394284    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:21.394514 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.394359    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:15:21.894341491 +0000 UTC m=+3.087519424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:21.394582 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.394546    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sgxj\" (UniqueName: \"kubernetes.io/projected/df23e468-1389-4fbc-9e50-06d5d97d3e1c-kube-api-access-9sgxj\") pod \"node-ca-tq265\" (UID: \"df23e468-1389-4fbc-9e50-06d5d97d3e1c\") " pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.395300 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.394917    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69m52\" (UniqueName: \"kubernetes.io/projected/c187ee75-0aff-41ff-be95-a05e6ca2cff3-kube-api-access-69m52\") pod \"iptables-alerter-v7hcf\" (UID: \"c187ee75-0aff-41ff-be95-a05e6ca2cff3\") " pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.397411 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.397386    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9gz\" (UniqueName: \"kubernetes.io/projected/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-kube-api-access-md9gz\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.397529 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.397484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qlp\" (UniqueName: \"kubernetes.io/projected/4163186b-2c9e-42d1-ba75-0f9d45be9ff8-kube-api-access-w5qlp\") pod \"multus-additional-cni-plugins-8plsn\" (UID: \"4163186b-2c9e-42d1-ba75-0f9d45be9ff8\") " pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.398126 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.398081    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdkh\" (UniqueName: \"kubernetes.io/projected/260c4dbc-58c6-4a0c-984d-9b8bce3f558a-kube-api-access-9xdkh\") pod \"tuned-l7w94\" (UID: \"260c4dbc-58c6-4a0c-984d-9b8bce3f558a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.398925 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.398902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kjw\" (UniqueName: \"kubernetes.io/projected/5c551ea5-6857-4c62-ab6c-dd0d9182a9dc-kube-api-access-q4kjw\") pod \"multus-gxvfx\" (UID: \"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc\") " pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.400078 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.400054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29jh\" (UniqueName: \"kubernetes.io/projected/a7d0a01d-6f21-4a48-b2bf-cff129a3d48c-kube-api-access-r29jh\") pod \"aws-ebs-csi-driver-node-z7fhr\" (UID: \"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.433109 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.432934    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:21.473883 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.473852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5bml"
Jul 24 21:15:21.482982 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.482960    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:21.506957 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.506934    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:21.515828 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.515809    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7hcf"
Jul 24 21:15:21.523861 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.523842    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr"
Jul 24 21:15:21.531759 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.531734    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8plsn"
Jul 24 21:15:21.539607 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.539586    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l7w94"
Jul 24 21:15:21.547433 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.547414    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxvfx"
Jul 24 21:15:21.554222 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.554201    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tq265"
Jul 24 21:15:21.888222 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.888182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:21.888344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.888225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:21.888344 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.888335    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:21.888448 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.888351    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:21.888448 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.888413    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:15:22.888390768 +0000 UTC m=+4.081568703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:21.888448 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.888434    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:15:22.888423279 +0000 UTC m=+4.081601201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:21.989499 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:21.989470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:21.989691 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.989628    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:21.989691 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.989651    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:21.989691 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.989666    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:21.989846 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:21.989753    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:15:22.989731999 +0000 UTC m=+4.182909924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:22.219470 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:22.219385    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:10:21 +0000 UTC" deadline="2028-01-25 04:17:46.331959024 +0000 UTC"
Jul 24 21:15:22.219470 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:22.219418    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13183h2m24.112545019s"
Jul 24 21:15:22.309198 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:22.309167    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:22.458190 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:22.458155    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980a987a_0aa5_4f5f_8c93_ab877c736e8e.slice/crio-d4c7aaf8e4607e5f6c7cad32e3dd38ab82c6f2ca94aed8ea8b51e8f446e68819 WatchSource:0}: Error finding container d4c7aaf8e4607e5f6c7cad32e3dd38ab82c6f2ca94aed8ea8b51e8f446e68819: Status 404 returned error can't find the container with id d4c7aaf8e4607e5f6c7cad32e3dd38ab82c6f2ca94aed8ea8b51e8f446e68819
Jul 24 21:15:22.490238 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:22.490213    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4163186b_2c9e_42d1_ba75_0f9d45be9ff8.slice/crio-6aa3413362f3c23ef84701f8a36b2fdb1042769706000f9252fcbbe51000c460 WatchSource:0}: Error finding container 6aa3413362f3c23ef84701f8a36b2fdb1042769706000f9252fcbbe51000c460: Status 404 returned error can't find the container with id 6aa3413362f3c23ef84701f8a36b2fdb1042769706000f9252fcbbe51000c460
Jul 24 21:15:22.490909 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:22.490880    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5452a9bc_1899_4ec2_ae0b_78039a6722e8.slice/crio-397c1fc9e165f9dc06960a500c3d773d9ea657a366b9a6443651bc440c29130d WatchSource:0}: Error finding container 397c1fc9e165f9dc06960a500c3d773d9ea657a366b9a6443651bc440c29130d: Status 404 returned error can't find the container with id 397c1fc9e165f9dc06960a500c3d773d9ea657a366b9a6443651bc440c29130d
Jul 24 21:15:22.491899 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:22.491714    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc187ee75_0aff_41ff_be95_a05e6ca2cff3.slice/crio-d8c42e5a9d471ef06e007ec3be881cbc72ab18a651a446dee0694f0d5b8378f5 WatchSource:0}: Error finding container d8c42e5a9d471ef06e007ec3be881cbc72ab18a651a446dee0694f0d5b8378f5: Status 404 returned error can't find the container with id d8c42e5a9d471ef06e007ec3be881cbc72ab18a651a446dee0694f0d5b8378f5
Jul 24 21:15:22.493114 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:22.493035    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260c4dbc_58c6_4a0c_984d_9b8bce3f558a.slice/crio-aeaa3fcc11799738ea5d38d648c8251ed805d3191e560355b8fb1202c606e682 WatchSource:0}: Error finding container aeaa3fcc11799738ea5d38d648c8251ed805d3191e560355b8fb1202c606e682: Status 404 returned error can't find the container with id aeaa3fcc11799738ea5d38d648c8251ed805d3191e560355b8fb1202c606e682
Jul 24 21:15:22.493963 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:22.493935    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392c898d_7923_488d_b0c1_1da116ad9ec0.slice/crio-3b4082d00d94835a25487b87383ae20b4da0b9063404897e8c033f8b8233b35c WatchSource:0}: Error finding container 3b4082d00d94835a25487b87383ae20b4da0b9063404897e8c033f8b8233b35c: Status 404 returned error can't find the container with id 3b4082d00d94835a25487b87383ae20b4da0b9063404897e8c033f8b8233b35c
Jul 24 21:15:22.896343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:22.896084    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:22.896507 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:22.896374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:22.896507 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.896282    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:22.896507 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.896477    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:15:24.896444061 +0000 UTC m=+6.089621993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:22.896662 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.896527    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:22.896662 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.896588    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:15:24.896570766 +0000 UTC m=+6.089748685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:22.997617 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:22.997587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:22.997836 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.997780    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:22.997836 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.997801    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:22.997836 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.997815    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:22.998009 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:22.997880    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:15:24.997860703 +0000 UTC m=+6.191038642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:23.219817 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.219705    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:10:21 +0000 UTC" deadline="2028-03-10 08:43:14.14615332 +0000 UTC"
Jul 24 21:15:23.219817 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.219766    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14267h27m50.926391687s"
Jul 24 21:15:23.324686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.324656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:23.324894 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:23.324812    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:23.325264 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.325240    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:23.325395 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:23.325371    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:23.325463 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.325449    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:23.325549 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:23.325529    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:23.358122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.357189    2585 generic.go:356] "Generic (PLEG): container finished" podID="1b1dd8d37b48cd7553e4d70ea9d781b1" containerID="ac0750492c849bc2963b6f1ef75c3b1c20fdda4cb61526b32b974aabf778cc36" exitCode=0
Jul 24 21:15:23.358122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.357279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal" event={"ID":"1b1dd8d37b48cd7553e4d70ea9d781b1","Type":"ContainerDied","Data":"ac0750492c849bc2963b6f1ef75c3b1c20fdda4cb61526b32b974aabf778cc36"}
Jul 24 21:15:23.364341 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.364280    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tq265" event={"ID":"df23e468-1389-4fbc-9e50-06d5d97d3e1c","Type":"ContainerStarted","Data":"b616b04a4511a193f96374150f295d41e261ffa02e51c3ffac0b3e70c52438e8"}
Jul 24 21:15:23.377367 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.377292    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5bml" event={"ID":"392c898d-7923-488d-b0c1-1da116ad9ec0","Type":"ContainerStarted","Data":"3b4082d00d94835a25487b87383ae20b4da0b9063404897e8c033f8b8233b35c"}
Jul 24 21:15:23.405388 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.405359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"d4c7aaf8e4607e5f6c7cad32e3dd38ab82c6f2ca94aed8ea8b51e8f446e68819"}
Jul 24 21:15:23.412420 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.412391    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal" event={"ID":"c1bea1286bfe6de533a5546f72ba9243","Type":"ContainerStarted","Data":"679fa165042a44a651547979aace5b7e46c6b8b4a472d80a12aea64c136b49e7"}
Jul 24 21:15:23.421594 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.421567    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr" event={"ID":"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c","Type":"ContainerStarted","Data":"64e79b9913a06cdb50283f41fb0e037d5f59627cf17ae0ff96467d0a4b96e927"}
Jul 24 21:15:23.423796 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.423771    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxvfx" event={"ID":"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc","Type":"ContainerStarted","Data":"0cb52da4fe8e74f5ce4aad1279d5a271fdc76ef229aae468b56f3d8924609006"}
Jul 24 21:15:23.429281 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.429231    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-243.ec2.internal" podStartSLOduration=2.429212806 podStartE2EDuration="2.429212806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:23.429020208 +0000 UTC m=+4.622198149" watchObservedRunningTime="2026-07-24 21:15:23.429212806 +0000 UTC m=+4.622390747"
Jul 24 21:15:23.443266 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.443239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l7w94" event={"ID":"260c4dbc-58c6-4a0c-984d-9b8bce3f558a","Type":"ContainerStarted","Data":"aeaa3fcc11799738ea5d38d648c8251ed805d3191e560355b8fb1202c606e682"}
Jul 24 21:15:23.446877 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.446851    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7hcf" event={"ID":"c187ee75-0aff-41ff-be95-a05e6ca2cff3","Type":"ContainerStarted","Data":"d8c42e5a9d471ef06e007ec3be881cbc72ab18a651a446dee0694f0d5b8378f5"}
Jul 24 21:15:23.467384 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.467353    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2hw84" event={"ID":"5452a9bc-1899-4ec2-ae0b-78039a6722e8","Type":"ContainerStarted","Data":"397c1fc9e165f9dc06960a500c3d773d9ea657a366b9a6443651bc440c29130d"}
Jul 24 21:15:23.471292 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:23.471225    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerStarted","Data":"6aa3413362f3c23ef84701f8a36b2fdb1042769706000f9252fcbbe51000c460"}
Jul 24 21:15:24.503034 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:24.503002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal" event={"ID":"1b1dd8d37b48cd7553e4d70ea9d781b1","Type":"ContainerStarted","Data":"cdc426e9783b4801a27752e4a0a1c84a9eb39b30040be28a269dbcf460bcd7d4"}
Jul 24 21:15:24.912669 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:24.912630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:24.912871 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:24.912704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:24.912871 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:24.912865    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:24.912996 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:24.912930    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:15:28.912911019 +0000 UTC m=+10.106088939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:24.913343 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:24.913321    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:24.913456 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:24.913378    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:15:28.913360857 +0000 UTC m=+10.106538779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:25.013434 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:25.013385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:25.013631 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.013612    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:25.013722 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.013632    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:25.013722 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.013647    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:25.013892 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.013733    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:15:29.013689544 +0000 UTC m=+10.206867466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:25.323879 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:25.323801    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:25.324039 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:25.323804    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:25.324039 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.323972    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:25.324275 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.324212    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:25.324275 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:25.324265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:25.324404 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:25.324316    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:27.324998 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:27.321866    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:27.324998 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:27.321997    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:27.324998 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:27.322517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:27.324998 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:27.322587    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:27.324998 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:27.322645    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:27.324998 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:27.322705    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:28.944572 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:28.944522    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:28.945008 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:28.944589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:28.945008 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:28.944780    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:28.945008 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:28.944832    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:15:36.944817622 +0000 UTC m=+18.137995537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:28.945169 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:28.945155    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:28.945225 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:28.945216    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:15:36.945205448 +0000 UTC m=+18.138383363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:29.045450 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:29.045405    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:29.045623 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.045569    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:29.045623 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.045592    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:29.045623 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.045605    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:29.045795 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.045683    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:15:37.04564786 +0000 UTC m=+18.238825777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:29.323621 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:29.323543    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:29.323801 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.323657    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:29.323801 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:29.323758    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:29.323907 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.323876    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:29.323964 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:29.323922    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:29.324020 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:29.323986    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:31.321857 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:31.321770    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:31.323833 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:31.323808    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:31.323981 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:31.323955    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:31.324057 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:31.324038    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:31.324240 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:31.324222    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:31.329477 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:31.322890    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:33.321722 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:33.321685    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:33.322185 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:33.321823    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:33.322185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:33.321700    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:33.322185 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:33.321937    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:33.322185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:33.321685    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:33.322185 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:33.322030    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:35.322540 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:35.322502    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:35.322540 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:35.322538    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:35.323053 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:35.322502    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:35.323053 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:35.322651    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:35.323053 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:35.322779    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:35.323053 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:35.322867    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:37.006301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:37.006268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:37.006301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:37.006306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:37.006864 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.006430    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:37.006864 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.006438    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:37.006864 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.006487    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:15:53.006466526 +0000 UTC m=+34.199644442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:37.006864 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.006501    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:15:53.006495238 +0000 UTC m=+34.199673153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:37.107194 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:37.107151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:37.107354 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.107301    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:37.107354 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.107324    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:37.107354 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.107333    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:37.107519 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.107374    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:15:53.107361698 +0000 UTC m=+34.300539613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:37.322255 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:37.322175    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:37.322255 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:37.322204    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:37.322452 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:37.322175    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:37.322452 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.322289    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:37.322452 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.322348    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:37.322615 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:37.322439    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:39.322534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:39.322499    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:39.322979 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:39.322605    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:39.322979 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:39.322636    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:39.322979 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:39.322649    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:39.322979 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:39.322729    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:39.322979 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:39.322831    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:41.321821 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.321704    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:41.321821 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:41.321806    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:41.322299 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.321867    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:41.322299 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:41.321933    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:41.322299 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.321981    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:41.322299 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:41.322055    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:41.546234 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.545938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr" event={"ID":"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c","Type":"ContainerStarted","Data":"b86ff6fe56c6961d9160b7b5fa1947d848217ee18f46ef8290b36c7ddf0c855a"}
Jul 24 21:15:41.549295 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.549082    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxvfx" event={"ID":"5c551ea5-6857-4c62-ab6c-dd0d9182a9dc","Type":"ContainerStarted","Data":"1321fbad8a9999f16dea016a0435cbca5190a372f86100e19eeeaad22d7f5ef0"}
Jul 24 21:15:41.551410 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.551364    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l7w94" event={"ID":"260c4dbc-58c6-4a0c-984d-9b8bce3f558a","Type":"ContainerStarted","Data":"d0ce0674ed0356a564ef3fe6524eeb6875bae49a33613a549a75ee46246931b8"}
Jul 24 21:15:41.554182 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.554153    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2hw84" event={"ID":"5452a9bc-1899-4ec2-ae0b-78039a6722e8","Type":"ContainerStarted","Data":"7dac52a0b80d5e64b7b8786470c36880896efdbca0abba5553f3a14c71babf27"}
Jul 24 21:15:41.559811 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.559719    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerStarted","Data":"03b0ae8c4520629e3c4bf4565b6a85cc4d04e6604e49d06c6bb9cafe0101fec5"}
Jul 24 21:15:41.562031 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.562005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tq265" event={"ID":"df23e468-1389-4fbc-9e50-06d5d97d3e1c","Type":"ContainerStarted","Data":"64761c7d30374dc5b2cb7d4e2d81db3a02a89f8a861ccb9694f650ae363c31ff"}
Jul 24 21:15:41.563897 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.563875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5bml" event={"ID":"392c898d-7923-488d-b0c1-1da116ad9ec0","Type":"ContainerStarted","Data":"92b5d36a2d17ddedaeb910d919a3cc5c5d570f4690acf8090d08c9ea3595b5f9"}
Jul 24 21:15:41.566123 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.566102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"74fd6182d79cf973cd22a8772b71875fe218f6aad70312992eda34198beec1f7"}
Jul 24 21:15:41.566221 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.566157    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"50962c601bb3c251a5bcf99e3414d7a21ececc8396b3cee1cf22060882995b28"}
Jul 24 21:15:41.571493 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.571393    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-243.ec2.internal" podStartSLOduration=20.571380198 podStartE2EDuration="20.571380198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:24.521082844 +0000 UTC m=+5.714260787" watchObservedRunningTime="2026-07-24 21:15:41.571380198 +0000 UTC m=+22.764558134"
Jul 24 21:15:41.571594 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.571482    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gxvfx" podStartSLOduration=2.7170418290000002 podStartE2EDuration="21.571471364s" totalImagesPullingTime="18.854429535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:41.570822113 +0000 UTC m=+22.764000051" watchObservedRunningTime="2026-07-24 21:15:41.571471364 +0000 UTC m=+22.764649302"
Jul 24 21:15:41.626995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.626941    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f5bml" podStartSLOduration=3.817328512 podStartE2EDuration="22.626924785s" totalImagesPullingTime="18.809596273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:41.60812567 +0000 UTC m=+22.801303606" watchObservedRunningTime="2026-07-24 21:15:41.626924785 +0000 UTC m=+22.820102721"
Jul 24 21:15:41.627207 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.627185    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l7w94" podStartSLOduration=2.8123107210000002 podStartE2EDuration="21.627177605s" totalImagesPullingTime="18.814866884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:41.627127797 +0000 UTC m=+22.820305735" watchObservedRunningTime="2026-07-24 21:15:41.627177605 +0000 UTC m=+22.820355595"
Jul 24 21:15:41.666347 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.666078    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2hw84" podStartSLOduration=11.665538255 podStartE2EDuration="22.666058089s" totalImagesPullingTime="11.000519834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:41.666021019 +0000 UTC m=+22.859198956" watchObservedRunningTime="2026-07-24 21:15:41.666058089 +0000 UTC m=+22.859236027"
Jul 24 21:15:41.666611 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.666571    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tq265" podStartSLOduration=2.858907144 podStartE2EDuration="21.666559055s" totalImagesPullingTime="18.807651911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:41.647560255 +0000 UTC m=+22.840738192" watchObservedRunningTime="2026-07-24 21:15:41.666559055 +0000 UTC m=+22.859737009"
Jul 24 21:15:41.730409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.730374    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:41.796486 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:41.796462    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:42.569101 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.569071    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7hcf" event={"ID":"c187ee75-0aff-41ff-be95-a05e6ca2cff3","Type":"ContainerStarted","Data":"f13717c1b9f7c40c103c3726b2912cddee6b268b0674bee17772e93849cf94d8"}
Jul 24 21:15:42.570619 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.570595    2585 generic.go:356] "Generic (PLEG): container finished" podID="4163186b-2c9e-42d1-ba75-0f9d45be9ff8" containerID="03b0ae8c4520629e3c4bf4565b6a85cc4d04e6604e49d06c6bb9cafe0101fec5" exitCode=0
Jul 24 21:15:42.570746 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.570692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerDied","Data":"03b0ae8c4520629e3c4bf4565b6a85cc4d04e6604e49d06c6bb9cafe0101fec5"}
Jul 24 21:15:42.573631 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.573610    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"ac368f51086e460968ca3db709a0f95cc264d6dcc5f691f51c721113d4122887"}
Jul 24 21:15:42.573746 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.573695    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"3cc8789bb5e8c1d669d0da2455a48aa2914dca2c02f304c3b8de22fae867cf6d"}
Jul 24 21:15:42.573746 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.573729    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"904378ddfa23ee71f39abbc129fef41fb977a08c138e754e55feafaaa4302834"}
Jul 24 21:15:42.573866 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.573756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"10cedf47cce90a247c56a3635ae4749289cf15ad9e64664abc755359598a9aff"}
Jul 24 21:15:42.585459 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.585420    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v7hcf" podStartSLOduration=3.772410677 podStartE2EDuration="22.585407465s" totalImagesPullingTime="18.812996788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:42.585161429 +0000 UTC m=+23.778339366" watchObservedRunningTime="2026-07-24 21:15:42.585407465 +0000 UTC m=+23.778585415"
Jul 24 21:15:42.705153 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.705127    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:42.727050 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.727029    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 21:15:42.806353 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.806291    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:42.906233 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:42.906209    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:43.007358 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.007275    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:43.061372 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.061302    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:43.236827 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.236715    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T21:15:42.727045581Z","UUID":"3c492f25-48d3-4926-9667-85206ef3efbf","Handler":null,"Name":"","Endpoint":""}
Jul 24 21:15:43.238489 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.238460    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 21:15:43.238615 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.238513    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 21:15:43.325938 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.325867    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:43.326136 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.325868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:43.326136 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:43.325985    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:43.326136 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.326021    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:43.326136 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:43.326117    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:43.326361 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:43.326208    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:43.578409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.578332    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr" event={"ID":"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c","Type":"ContainerStarted","Data":"b498c45a68d3786e802a319f2959520e0baf346ca50eb1313bc4c697bd06f286"}
Jul 24 21:15:43.779407 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.779379    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:43.779965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:43.779948    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:44.584964 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:44.584871    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"e359d5feca07b4db2990c767861fc01bab212c58191cf5324bc322b6f424846f"}
Jul 24 21:15:44.587746 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:44.587718    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr" event={"ID":"a7d0a01d-6f21-4a48-b2bf-cff129a3d48c","Type":"ContainerStarted","Data":"510adb83161a3df48c53891df4086cf1cd8b88642004a30e9082dcd7ad993976"}
Jul 24 21:15:44.587876 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:44.587772    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:44.588146 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:44.588127    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2hw84"
Jul 24 21:15:44.624286 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:44.624245    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7fhr" podStartSLOduration=3.750620911 podStartE2EDuration="24.624232925s" totalImagesPullingTime="20.873612014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:44.607921692 +0000 UTC m=+25.801099655" watchObservedRunningTime="2026-07-24 21:15:44.624232925 +0000 UTC m=+25.817410862"
Jul 24 21:15:44.673850 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:44.673819    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:45.325786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:45.325759    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:45.325961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:45.325903    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:45.325961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:45.325923    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:45.325961 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:45.325919    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:45.326132 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:45.326004    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:45.326132 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:45.326072    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:46.595473 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:46.595298    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" event={"ID":"980a987a-0aa5-4f5f-8c93-ab877c736e8e","Type":"ContainerStarted","Data":"7370561c6f34ee7c006cffda9b0a810c2971de334c0a45e6b33a6a1232afe779"}
Jul 24 21:15:46.596180 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:46.595713    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:46.596180 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:46.595779    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:46.617208 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:46.617009    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:46.625215 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:46.625034    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n" podStartSLOduration=8.732242646 podStartE2EDuration="27.625017629s" totalImagesPullingTime="18.892774983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:46.623540242 +0000 UTC m=+27.816718180" watchObservedRunningTime="2026-07-24 21:15:46.625017629 +0000 UTC m=+27.818195566"
Jul 24 21:15:47.321930 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.321899    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:47.321930 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.321918    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:47.322150 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.321919    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:47.322150 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:47.322032    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:47.322150 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:47.322102    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:47.322296 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:47.322192    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:47.598810 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.598737    2585 generic.go:356] "Generic (PLEG): container finished" podID="4163186b-2c9e-42d1-ba75-0f9d45be9ff8" containerID="d3988224d942bf7b8eca839f56dc94fbc83f2d08ca7d3360832517cb802f0d4e" exitCode=0
Jul 24 21:15:47.599499 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.598828    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerDied","Data":"d3988224d942bf7b8eca839f56dc94fbc83f2d08ca7d3360832517cb802f0d4e"}
Jul 24 21:15:47.599499 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.599325    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:47.614602 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:47.614582    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:15:48.537440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.537218    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7kk4m"]
Jul 24 21:15:48.537583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.537543    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:48.537703 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:48.537660    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:48.545689 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.545592    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6hjf"]
Jul 24 21:15:48.545792 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.545695    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:48.545905 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:48.545815    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:48.546167 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.546148    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5sl4p"]
Jul 24 21:15:48.546231 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.546219    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:48.546306 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:48.546291    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:48.550437 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.550418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:48.603118 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.603091    2585 generic.go:356] "Generic (PLEG): container finished" podID="4163186b-2c9e-42d1-ba75-0f9d45be9ff8" containerID="cc11b11337fd1f6160cc160d3d06b629287ec64eea217b81f02350fa83425f74" exitCode=0
Jul 24 21:15:48.603476 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.603173    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerDied","Data":"cc11b11337fd1f6160cc160d3d06b629287ec64eea217b81f02350fa83425f74"}
Jul 24 21:15:48.606902 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.606884    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:48.667728 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:48.667703    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:49.522984 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:49.522953    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:49.580702 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:49.580659    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:50.322365 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:50.322334    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:50.322764 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:50.322416    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:50.322764 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:50.322422    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:50.322764 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:50.322420    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:50.322764 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:50.322500    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:50.322764 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:50.322558    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:50.609255 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:50.609182    2585 generic.go:356] "Generic (PLEG): container finished" podID="4163186b-2c9e-42d1-ba75-0f9d45be9ff8" containerID="2d3255ad40514dfd8a0e9bf3d539d2f8953553bb8d53eb1a4bd38b369438bd43" exitCode=0
Jul 24 21:15:50.609255 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:50.609245    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerDied","Data":"2d3255ad40514dfd8a0e9bf3d539d2f8953553bb8d53eb1a4bd38b369438bd43"}
Jul 24 21:15:51.245438 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:51.245348    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:51.346210 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:51.346168    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.322547 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:52.322516    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:52.322749 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:52.322622    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:52.322749 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:52.322631    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:52.322749 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:52.322651    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:52.322898 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:52.322751    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:52.322898 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:52.322854    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:53.028934 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:53.028897    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:53.028934 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:53.028937    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:53.029446 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.029072    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:53.029446 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.029088    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:53.029446 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.029136    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret podName:67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf nodeName:}" failed. No retries permitted until 2026-07-24 21:16:25.029119031 +0000 UTC m=+66.222296945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret") pod "global-pull-secret-syncer-7kk4m" (UID: "67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:53.029446 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.029157    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs podName:694177d1-b8fa-43ec-9ad0-dbd7aa96111f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:25.029146519 +0000 UTC m=+66.222324438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs") pod "network-metrics-daemon-5sl4p" (UID: "694177d1-b8fa-43ec-9ad0-dbd7aa96111f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:53.129557 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:53.129514    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:53.129770 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.129750    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:53.129770 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.129774    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:53.129890 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.129785    2585 projected.go:196] Error preparing data for projected volume kube-api-access-qbptj for pod openshift-network-diagnostics/network-check-target-n6hjf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:53.129890 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:53.129846    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj podName:2dae7df5-d719-4b5b-83c5-be9c6af6b24c nodeName:}" failed. No retries permitted until 2026-07-24 21:16:25.129828635 +0000 UTC m=+66.323006558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qbptj" (UniqueName: "kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj") pod "network-check-target-n6hjf" (UID: "2dae7df5-d719-4b5b-83c5-be9c6af6b24c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:53.845399 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:53.845200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:53.895502 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:53.895468    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:54.322247 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:54.322215    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:54.322598 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:54.322352    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kk4m" podUID="67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf"
Jul 24 21:15:54.322802 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:54.322784    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:54.322930 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:54.322903    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5sl4p" podUID="694177d1-b8fa-43ec-9ad0-dbd7aa96111f"
Jul 24 21:15:54.323044 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:54.322962    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:54.323044 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:54.323032    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n6hjf" podUID="2dae7df5-d719-4b5b-83c5-be9c6af6b24c"
Jul 24 21:15:55.175567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.175542    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-243.ec2.internal" event="NodeReady"
Jul 24 21:15:55.175711 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.175632    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 21:15:55.209923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.209894    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75c7d9b5f6-55qpl"]
Jul 24 21:15:55.211449 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.211433    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.215346 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.215271    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 21:15:55.215659 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.215338    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zdcdd\""
Jul 24 21:15:55.215799 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.215602    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 21:15:55.215799 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.215678    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 21:15:55.220209 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.220190    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 21:15:55.225498 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.225479    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vkrgq"]
Jul 24 21:15:55.227154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.227139    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.228903 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.228872    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75c7d9b5f6-55qpl"]
Jul 24 21:15:55.229606 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.229584    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 21:15:55.229718 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.229632    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fczl6"]
Jul 24 21:15:55.229718 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.229657    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t26tv\""
Jul 24 21:15:55.232159 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.232143    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.234407 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.234391    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 21:15:55.235070 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.235051    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 21:15:55.235406 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.235387    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 21:15:55.235493 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.235387    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jpr64\""
Jul 24 21:15:55.235493 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.235477    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 21:15:55.243999 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.243980    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vkrgq"]
Jul 24 21:15:55.247715 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.247656    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fczl6"]
Jul 24 21:15:55.347392 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347359    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-certificates\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347392 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-bound-sa-token\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347410    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p68p\" (UniqueName: \"kubernetes.io/projected/37d7e99c-5d52-4c53-a7a2-e46f3910261a-kube-api-access-9p68p\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347427    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347448    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vzd\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-kube-api-access-h8vzd\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347506    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-installation-pull-secrets\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7e99c-5d52-4c53-a7a2-e46f3910261a-tmp-dir\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347573    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551c542a-9f19-47f8-b41e-2dd2f21f674d-cert\") pod \"ingress-canary-fczl6\" (UID: \"551c542a-9f19-47f8-b41e-2dd2f21f674d\") " pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347617    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347634    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7e99c-5d52-4c53-a7a2-e46f3910261a-config-volume\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-image-registry-private-configuration\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347675    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-trusted-ca\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347693    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8p5\" (UniqueName: \"kubernetes.io/projected/551c542a-9f19-47f8-b41e-2dd2f21f674d-kube-api-access-9z8p5\") pod \"ingress-canary-fczl6\" (UID: \"551c542a-9f19-47f8-b41e-2dd2f21f674d\") " pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.347895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.347709    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc3f06ac-a6e1-409e-a719-eaf3da600093-ca-trust-extracted\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.358239 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.358218    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:55.448513 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448424    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-installation-pull-secrets\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.448513 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7e99c-5d52-4c53-a7a2-e46f3910261a-tmp-dir\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.448513 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551c542a-9f19-47f8-b41e-2dd2f21f674d-cert\") pod \"ingress-canary-fczl6\" (UID: \"551c542a-9f19-47f8-b41e-2dd2f21f674d\") " pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.448785 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.448785 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7e99c-5d52-4c53-a7a2-e46f3910261a-config-volume\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.448785 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-image-registry-private-configuration\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.448785 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448612    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-trusted-ca\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.448785 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448638    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8p5\" (UniqueName: \"kubernetes.io/projected/551c542a-9f19-47f8-b41e-2dd2f21f674d-kube-api-access-9z8p5\") pod \"ingress-canary-fczl6\" (UID: \"551c542a-9f19-47f8-b41e-2dd2f21f674d\") " pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.448785 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.448732    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc3f06ac-a6e1-409e-a719-eaf3da600093-ca-trust-extracted\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.448813    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls podName:37d7e99c-5d52-4c53-a7a2-e46f3910261a nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.948792695 +0000 UTC m=+37.141970625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls") pod "dns-default-vkrgq" (UID: "37d7e99c-5d52-4c53-a7a2-e46f3910261a") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448865    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-certificates\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-bound-sa-token\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p68p\" (UniqueName: \"kubernetes.io/projected/37d7e99c-5d52-4c53-a7a2-e46f3910261a-kube-api-access-9p68p\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448939    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d7e99c-5d52-4c53-a7a2-e46f3910261a-tmp-dir\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448939    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.448994    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vzd\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-kube-api-access-h8vzd\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.449029    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:55.449069 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.449041    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c7d9b5f6-55qpl: secret "image-registry-tls" not found
Jul 24 21:15:55.449533 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.449092    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls podName:cc3f06ac-a6e1-409e-a719-eaf3da600093 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.949076352 +0000 UTC m=+37.142254308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls") pod "image-registry-75c7d9b5f6-55qpl" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093") : secret "image-registry-tls" not found
Jul 24 21:15:55.449533 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.449283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37d7e99c-5d52-4c53-a7a2-e46f3910261a-config-volume\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.449533 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.449471    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc3f06ac-a6e1-409e-a719-eaf3da600093-ca-trust-extracted\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.449835 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.449810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-certificates\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.450380 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.450354    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-trusted-ca\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.452933 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.452911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-installation-pull-secrets\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.453008 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.452938    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-image-registry-private-configuration\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.453008 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.452987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551c542a-9f19-47f8-b41e-2dd2f21f674d-cert\") pod \"ingress-canary-fczl6\" (UID: \"551c542a-9f19-47f8-b41e-2dd2f21f674d\") " pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.458446 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.458421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:55.460772 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.460753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p68p\" (UniqueName: \"kubernetes.io/projected/37d7e99c-5d52-4c53-a7a2-e46f3910261a-kube-api-access-9p68p\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.460890 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.460870    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-bound-sa-token\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.461176 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.461157    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8p5\" (UniqueName: \"kubernetes.io/projected/551c542a-9f19-47f8-b41e-2dd2f21f674d-kube-api-access-9z8p5\") pod \"ingress-canary-fczl6\" (UID: \"551c542a-9f19-47f8-b41e-2dd2f21f674d\") " pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.461328 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.461310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vzd\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-kube-api-access-h8vzd\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.542102 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.542072    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fczl6"
Jul 24 21:15:55.626965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.626930    2585 generic.go:356] "Generic (PLEG): container finished" podID="4163186b-2c9e-42d1-ba75-0f9d45be9ff8" containerID="848d57061f104aeedf39a61c179f259f4c613de9c285c117692496a2d1082903" exitCode=0
Jul 24 21:15:55.627086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.627019    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerDied","Data":"848d57061f104aeedf39a61c179f259f4c613de9c285c117692496a2d1082903"}
Jul 24 21:15:55.678216 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.678182    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:55.678385 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.678368    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fczl6"]
Jul 24 21:15:55.691760 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:55.691596    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551c542a_9f19_47f8_b41e_2dd2f21f674d.slice/crio-f78ca2007b1bf8b69bb48b51be1705d63d3ab7c46c41d755db95cecc52024075 WatchSource:0}: Error finding container f78ca2007b1bf8b69bb48b51be1705d63d3ab7c46c41d755db95cecc52024075: Status 404 returned error can't find the container with id f78ca2007b1bf8b69bb48b51be1705d63d3ab7c46c41d755db95cecc52024075
Jul 24 21:15:55.953538 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.953498    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:55.953711 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:55.953561    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:55.953711 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.953708    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:55.953793 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.953705    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:55.953793 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.953721    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c7d9b5f6-55qpl: secret "image-registry-tls" not found
Jul 24 21:15:55.953793 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.953788    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls podName:cc3f06ac-a6e1-409e-a719-eaf3da600093 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:56.953768249 +0000 UTC m=+38.146946179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls") pod "image-registry-75c7d9b5f6-55qpl" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093") : secret "image-registry-tls" not found
Jul 24 21:15:55.953894 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:55.953801    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls podName:37d7e99c-5d52-4c53-a7a2-e46f3910261a nodeName:}" failed. No retries permitted until 2026-07-24 21:15:56.953795579 +0000 UTC m=+38.146973493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls") pod "dns-default-vkrgq" (UID: "37d7e99c-5d52-4c53-a7a2-e46f3910261a") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:56.322030 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.321988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:15:56.322184 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.321988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:15:56.322259 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.321988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:15:56.325655 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.325615    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:15:56.325858 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.325738    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:15:56.326861 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.326792    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:15:56.326861 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.326804    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:15:56.326861 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.326829    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66lqr\""
Jul 24 21:15:56.327115 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.326895    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7jm8m\""
Jul 24 21:15:56.632213 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.632171    2585 generic.go:356] "Generic (PLEG): container finished" podID="4163186b-2c9e-42d1-ba75-0f9d45be9ff8" containerID="5ec3af9f3c1ac55a2302a144d1ffa98b61549066e3b37059485ae6e3ca234f53" exitCode=0
Jul 24 21:15:56.633133 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.632258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerDied","Data":"5ec3af9f3c1ac55a2302a144d1ffa98b61549066e3b37059485ae6e3ca234f53"}
Jul 24 21:15:56.634281 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.634246    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fczl6" event={"ID":"551c542a-9f19-47f8-b41e-2dd2f21f674d","Type":"ContainerStarted","Data":"f78ca2007b1bf8b69bb48b51be1705d63d3ab7c46c41d755db95cecc52024075"}
Jul 24 21:15:56.962435 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.962356    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:56.962435 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:56.962423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:56.962633 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:56.962509    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:56.962633 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:56.962536    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:56.962633 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:56.962550    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c7d9b5f6-55qpl: secret "image-registry-tls" not found
Jul 24 21:15:56.962633 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:56.962589    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls podName:37d7e99c-5d52-4c53-a7a2-e46f3910261a nodeName:}" failed. No retries permitted until 2026-07-24 21:15:58.962568285 +0000 UTC m=+40.155746216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls") pod "dns-default-vkrgq" (UID: "37d7e99c-5d52-4c53-a7a2-e46f3910261a") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:56.962633 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:56.962612    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls podName:cc3f06ac-a6e1-409e-a719-eaf3da600093 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:58.962601416 +0000 UTC m=+40.155779331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls") pod "image-registry-75c7d9b5f6-55qpl" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093") : secret "image-registry-tls" not found
Jul 24 21:15:57.063952 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.063918    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.227853 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.227776    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xc5zd"]
Jul 24 21:15:57.229733 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.229714    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.232533 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.232515    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 21:15:57.232644 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.232536    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cfdvq\""
Jul 24 21:15:57.233794 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.233775    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 21:15:57.233930 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.233781    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 21:15:57.233930 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.233909    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 21:15:57.248780 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.248761    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xc5zd"]
Jul 24 21:15:57.256215 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.256194    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.322451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.322417    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.364934 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.364901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b90561-8565-4a7f-a290-73b7dd0a1f60-data-volume\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.364934 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.364929    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.365115 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.365053    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b90561-8565-4a7f-a290-73b7dd0a1f60-crio-socket\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.365115 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.365086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.365204 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.365129    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtpb\" (UniqueName: \"kubernetes.io/projected/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-api-access-nvtpb\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466308 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b90561-8565-4a7f-a290-73b7dd0a1f60-crio-socket\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466469 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466318    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466469 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466352    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtpb\" (UniqueName: \"kubernetes.io/projected/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-api-access-nvtpb\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466469 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466389    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b90561-8565-4a7f-a290-73b7dd0a1f60-data-volume\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466469 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466407    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466708 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.466492    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:57.466708 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70b90561-8565-4a7f-a290-73b7dd0a1f60-crio-socket\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.466708 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.466526    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:57.466708 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.466558    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:57.966538282 +0000 UTC m=+39.159716215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:57.466708 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.466587    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:57.966575817 +0000 UTC m=+39.159753733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:57.466965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.466834    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70b90561-8565-4a7f-a290-73b7dd0a1f60-data-volume\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.477508 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.477484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtpb\" (UniqueName: \"kubernetes.io/projected/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-api-access-nvtpb\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.519543 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.519466    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.575755 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.575725    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.640474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.640445    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8plsn" event={"ID":"4163186b-2c9e-42d1-ba75-0f9d45be9ff8","Type":"ContainerStarted","Data":"3f03db01d872b9e4e178fd22f36d8a053ae366e3540348345a38829fbe87e0af"}
Jul 24 21:15:57.668821 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.668766    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8plsn" podStartSLOduration=9.081561548 podStartE2EDuration="37.668740333s" totalImagesPullingTime="28.587178785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:57.665609999 +0000 UTC m=+38.858787984" watchObservedRunningTime="2026-07-24 21:15:57.668740333 +0000 UTC m=+38.861918272"
Jul 24 21:15:57.783572 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.783494    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.840961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.840914    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.969763 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.969728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.969908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.969810    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:57.969908 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.969847    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:57.969981 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.969916    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:57.969981 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.969918    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:58.969901359 +0000 UTC m=+40.163079274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:57.969981 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:57.969965    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:58.969952398 +0000 UTC m=+40.163130312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:57.971261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.971238    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.994330 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:57.994302    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"]
Jul 24 21:15:58.013151 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.013130    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"]
Jul 24 21:15:58.013266 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.013243    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"
Jul 24 21:15:58.015940 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.015918    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mm5pj\""
Jul 24 21:15:58.016183 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.016164    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:58.016243 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.016167    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 24 21:15:58.037596 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.037541    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.096613 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.096575    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.171618 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.171598    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccb9\" (UniqueName: \"kubernetes.io/projected/be4d127f-5b1f-412b-8ffe-dc2b2e940908-kube-api-access-gccb9\") pod \"migrator-6865c9967-lqpct\" (UID: \"be4d127f-5b1f-412b-8ffe-dc2b2e940908\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"
Jul 24 21:15:58.272634 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.272478    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccb9\" (UniqueName: \"kubernetes.io/projected/be4d127f-5b1f-412b-8ffe-dc2b2e940908-kube-api-access-gccb9\") pod \"migrator-6865c9967-lqpct\" (UID: \"be4d127f-5b1f-412b-8ffe-dc2b2e940908\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"
Jul 24 21:15:58.283232 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.283210    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccb9\" (UniqueName: \"kubernetes.io/projected/be4d127f-5b1f-412b-8ffe-dc2b2e940908-kube-api-access-gccb9\") pod \"migrator-6865c9967-lqpct\" (UID: \"be4d127f-5b1f-412b-8ffe-dc2b2e940908\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"
Jul 24 21:15:58.322288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.322271    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"
Jul 24 21:15:58.373819 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.373787    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.462392 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.462340    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct"]
Jul 24 21:15:58.464826 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:58.464796    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4d127f_5b1f_412b_8ffe_dc2b2e940908.slice/crio-4fc242884a5f5dedff4168c7805ce1c626c50d07e587b9d8ad25b5674374f92b WatchSource:0}: Error finding container 4fc242884a5f5dedff4168c7805ce1c626c50d07e587b9d8ad25b5674374f92b: Status 404 returned error can't find the container with id 4fc242884a5f5dedff4168c7805ce1c626c50d07e587b9d8ad25b5674374f92b
Jul 24 21:15:58.465211 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.465190    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.644258 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.644225    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct" event={"ID":"be4d127f-5b1f-412b-8ffe-dc2b2e940908","Type":"ContainerStarted","Data":"4fc242884a5f5dedff4168c7805ce1c626c50d07e587b9d8ad25b5674374f92b"}
Jul 24 21:15:58.645661 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.645632    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fczl6" event={"ID":"551c542a-9f19-47f8-b41e-2dd2f21f674d","Type":"ContainerStarted","Data":"ba674277a74e981482e7b707bf51f35e5b28f2b539ea400ae5b1a18e4342fdf7"}
Jul 24 21:15:58.664883 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.664838    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fczl6" podStartSLOduration=1.073323667 podStartE2EDuration="3.664823539s" totalImagesPullingTime="2.591499872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:58.664129974 +0000 UTC m=+39.857307913" watchObservedRunningTime="2026-07-24 21:15:58.664823539 +0000 UTC m=+39.858001492"
Jul 24 21:15:58.801019 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.800987    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5bml_392c898d-7923-488d-b0c1-1da116ad9ec0/dns-node-resolver/0.log"
Jul 24 21:15:58.977942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.977822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:15:58.977942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.977910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.977985    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.977991    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978014    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c7d9b5f6-55qpl: secret "image-registry-tls" not found
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:58.978022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978051    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978071    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls podName:cc3f06ac-a6e1-409e-a719-eaf3da600093 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:02.978049452 +0000 UTC m=+44.171227367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls") pod "image-registry-75c7d9b5f6-55qpl" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093") : secret "image-registry-tls" not found
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978105    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:00.978088061 +0000 UTC m=+42.171265977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : configmap "kube-rbac-proxy" not found
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978126    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:58.978137 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978134    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:58.978580 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978166    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:00.978151871 +0000 UTC m=+42.171329787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:15:58.978580 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:15:58.978182    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls podName:37d7e99c-5d52-4c53-a7a2-e46f3910261a nodeName:}" failed. No retries permitted until 2026-07-24 21:16:02.978174791 +0000 UTC m=+44.171352710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls") pod "dns-default-vkrgq" (UID: "37d7e99c-5d52-4c53-a7a2-e46f3910261a") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:59.370886 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.370855    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4b6lw"]
Jul 24 21:15:59.397817 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.397782    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4b6lw"]
Jul 24 21:15:59.397950 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.397889    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.400765 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.400736    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:59.400765 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.400783    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 24 21:15:59.400982 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.400929    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 24 21:15:59.400982 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.400944    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 24 21:15:59.401153 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.401133    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 24 21:15:59.402131 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.402109    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wmh2p\""
Jul 24 21:15:59.461397 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.461358    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:59.583241 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.583211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-signing-key\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.583241 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.583242    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-signing-cabundle\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.583449 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.583272    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9957v\" (UniqueName: \"kubernetes.io/projected/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-kube-api-access-9957v\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.683923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.683860    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-signing-key\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.683923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.683887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-signing-cabundle\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.683923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.683916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9957v\" (UniqueName: \"kubernetes.io/projected/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-kube-api-access-9957v\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.684520 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.684492    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-signing-cabundle\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.686413 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.686388    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-signing-key\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.693118 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.693094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9957v\" (UniqueName: \"kubernetes.io/projected/8a1f57e4-376d-41e9-b1a8-7bbf39214ca2-kube-api-access-9957v\") pod \"service-ca-658d7b7c78-4b6lw\" (UID: \"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2\") " pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.708955 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.708933    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw"
Jul 24 21:15:59.865968 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.865947    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-4b6lw"]
Jul 24 21:15:59.883184 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.883162    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"]
Jul 24 21:15:59.896794 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.896775    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:15:59.898835 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.898807    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"]
Jul 24 21:15:59.899768 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.899747    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 24 21:15:59.899888 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.899792    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 24 21:15:59.899888 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.899798    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-x5zbw\""
Jul 24 21:15:59.942131 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:15:59.942075    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1f57e4_376d_41e9_b1a8_7bbf39214ca2.slice/crio-f67283d1dba8f3763571ead777475b672755a9d64886f2058af5e38325a5c624 WatchSource:0}: Error finding container f67283d1dba8f3763571ead777475b672755a9d64886f2058af5e38325a5c624: Status 404 returned error can't find the container with id f67283d1dba8f3763571ead777475b672755a9d64886f2058af5e38325a5c624
Jul 24 21:15:59.987004 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.986985    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:15:59.987093 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.987013    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:15:59.998709 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:15:59.998695    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tq265_df23e468-1389-4fbc-9e50-06d5d97d3e1c/node-ca/0.log"
Jul 24 21:16:00.018895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.018873    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.087402 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.087384    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:00.087504 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.087410    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:00.087567 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.087512    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 21:16:00.087567 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.087555    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert podName:cbefa37b-bdef-4291-8ca3-2271f8f5b02d nodeName:}" failed. No retries permitted until 2026-07-24 21:16:00.587543393 +0000 UTC m=+41.780721308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qpt4v" (UID: "cbefa37b-bdef-4291-8ca3-2271f8f5b02d") : secret "networking-console-plugin-cert" not found
Jul 24 21:16:00.088045 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.088025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:00.122805 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.122781    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.259511 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.259446    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.358897 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.358866    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.412738 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.412713    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fczl6_551c542a-9f19-47f8-b41e-2dd2f21f674d/serve-healthcheck-canary/0.log"
Jul 24 21:16:00.592336 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.592255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:00.592477 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.592427    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 21:16:00.592540 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.592514    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert podName:cbefa37b-bdef-4291-8ca3-2271f8f5b02d nodeName:}" failed. No retries permitted until 2026-07-24 21:16:01.592483657 +0000 UTC m=+42.785661573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qpt4v" (UID: "cbefa37b-bdef-4291-8ca3-2271f8f5b02d") : secret "networking-console-plugin-cert" not found
Jul 24 21:16:00.652887 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.652853    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct" event={"ID":"be4d127f-5b1f-412b-8ffe-dc2b2e940908","Type":"ContainerStarted","Data":"be774943bb039bcaa2a690eba24efdaa24fa900cc43c7194c8a3db240b5873b6"}
Jul 24 21:16:00.653029 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.652914    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct" event={"ID":"be4d127f-5b1f-412b-8ffe-dc2b2e940908","Type":"ContainerStarted","Data":"775796679ffcea1a68da633cba3a87b828bee23ec34cca3449c661ec77496a7b"}
Jul 24 21:16:00.654126 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.654099    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw" event={"ID":"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2","Type":"ContainerStarted","Data":"f67283d1dba8f3763571ead777475b672755a9d64886f2058af5e38325a5c624"}
Jul 24 21:16:00.659796 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.659767    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:00.679230 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.679187    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lqpct" podStartSLOduration=2.177835403 podStartE2EDuration="3.679172013s" totalImagesPullingTime="1.50133661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:00.679109489 +0000 UTC m=+41.872287447" watchObservedRunningTime="2026-07-24 21:16:00.679172013 +0000 UTC m=+41.872349950"
Jul 24 21:16:00.995487 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.995446    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:00.995902 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:00.995550    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:00.995902 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.995579    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:00.995902 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.995650    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:00.995902 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.995656    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:04.995641132 +0000 UTC m=+46.188819046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:00.995902 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:00.995738    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:04.995720265 +0000 UTC m=+46.188898184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:01.601288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:01.601257    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:01.601467 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:01.601405    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 21:16:01.601516 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:01.601470    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert podName:cbefa37b-bdef-4291-8ca3-2271f8f5b02d nodeName:}" failed. No retries permitted until 2026-07-24 21:16:03.60145508 +0000 UTC m=+44.794632999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qpt4v" (UID: "cbefa37b-bdef-4291-8ca3-2271f8f5b02d") : secret "networking-console-plugin-cert" not found
Jul 24 21:16:02.661433 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:02.661396    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw" event={"ID":"8a1f57e4-376d-41e9-b1a8-7bbf39214ca2","Type":"ContainerStarted","Data":"77d58875deb5c6d63c771f4e71673a4a178ca01f53d38d50372cb34c3b05de3b"}
Jul 24 21:16:02.682946 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:02.682339    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-4b6lw" podStartSLOduration=1.652025885 podStartE2EDuration="3.682320747s" totalImagesPullingTime="2.030294862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:02.681002889 +0000 UTC m=+43.874180826" watchObservedRunningTime="2026-07-24 21:16:02.682320747 +0000 UTC m=+43.875498687"
Jul 24 21:16:03.011605 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.011530    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:16:03.011745 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.011616    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:03.011745 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.011661    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:16:03.011821 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.011749    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls podName:37d7e99c-5d52-4c53-a7a2-e46f3910261a nodeName:}" failed. No retries permitted until 2026-07-24 21:16:11.01172871 +0000 UTC m=+52.204906639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls") pod "dns-default-vkrgq" (UID: "37d7e99c-5d52-4c53-a7a2-e46f3910261a") : secret "dns-default-metrics-tls" not found
Jul 24 21:16:03.011821 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.011751    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 21:16:03.011821 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.011766    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c7d9b5f6-55qpl: secret "image-registry-tls" not found
Jul 24 21:16:03.011821 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.011801    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls podName:cc3f06ac-a6e1-409e-a719-eaf3da600093 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:11.011790287 +0000 UTC m=+52.204968202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls") pod "image-registry-75c7d9b5f6-55qpl" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093") : secret "image-registry-tls" not found
Jul 24 21:16:03.264723 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.264621    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.360145 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.360112    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.551598 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.551493    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.617603 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.617570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:03.617793 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.617754    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 21:16:03.617865 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:03.617831    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert podName:cbefa37b-bdef-4291-8ca3-2271f8f5b02d nodeName:}" failed. No retries permitted until 2026-07-24 21:16:07.617812434 +0000 UTC m=+48.810990354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qpt4v" (UID: "cbefa37b-bdef-4291-8ca3-2271f8f5b02d") : secret "networking-console-plugin-cert" not found
Jul 24 21:16:03.651818 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.651786    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.709698 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:03.709657    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.160267 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:04.160235    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.347297 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:04.347269    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.418629 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:04.418569    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.815268 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:04.815200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.027513 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:05.027481    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:05.027686 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:05.027544    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:05.027686 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:05.027646    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:05.027686 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:05.027679    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:05.027807 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:05.027733    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:13.027718716 +0000 UTC m=+54.220896631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:05.027807 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:05.027744    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:13.027738816 +0000 UTC m=+54.220916731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:05.550318 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:05.550281    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.758107 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:05.758075    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.951249 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:05.951209    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:06.143922 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:06.143890    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:06.352469 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:06.352392    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:06.950713 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:06.950684    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.090863 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:07.090836    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.547472 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:07.547436    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.648549 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:07.648519    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:07.648720 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:07.648621    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 24 21:16:07.648720 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:07.648690    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert podName:cbefa37b-bdef-4291-8ca3-2271f8f5b02d nodeName:}" failed. No retries permitted until 2026-07-24 21:16:15.648659483 +0000 UTC m=+56.841837398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qpt4v" (UID: "cbefa37b-bdef-4291-8ca3-2271f8f5b02d") : secret "networking-console-plugin-cert" not found
Jul 24 21:16:07.753097 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:07.753066    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:07.955512 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:07.955478    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:08.021556 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:08.021524    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:08.150639 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:08.150603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:08.355030 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:08.354951    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.469455 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:09.469420    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.570862 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:09.570831    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.757645 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:09.757566    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:09.947969 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:09.947938    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.147738 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:10.147712    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.348278 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:10.348245    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:10.450132 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:10.450064    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:11.076241 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.076214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:16:11.076637 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.076264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:11.078648 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.078627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37d7e99c-5d52-4c53-a7a2-e46f3910261a-metrics-tls\") pod \"dns-default-vkrgq\" (UID: \"37d7e99c-5d52-4c53-a7a2-e46f3910261a\") " pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:16:11.078783 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.078686    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"image-registry-75c7d9b5f6-55qpl\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") " pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:11.123784 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.123755    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:11.137898 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.137864    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:16:11.157999 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.157967    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:11.291193 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.291171    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75c7d9b5f6-55qpl"]
Jul 24 21:16:11.293359 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:11.293336    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3f06ac_a6e1_409e_a719_eaf3da600093.slice/crio-e1d91be2d410676f8fe6e280e8423b9e3565191a557ef9a175ab1ea15b94dc7d WatchSource:0}: Error finding container e1d91be2d410676f8fe6e280e8423b9e3565191a557ef9a175ab1ea15b94dc7d: Status 404 returned error can't find the container with id e1d91be2d410676f8fe6e280e8423b9e3565191a557ef9a175ab1ea15b94dc7d
Jul 24 21:16:11.302731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.302710    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vkrgq"]
Jul 24 21:16:11.305129 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:11.305107    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d7e99c_5d52_4c53_a7a2_e46f3910261a.slice/crio-d336b5999c7868b61dd75e0106e32860e07144c4440eafd7237ff3cb9d1409f4 WatchSource:0}: Error finding container d336b5999c7868b61dd75e0106e32860e07144c4440eafd7237ff3cb9d1409f4: Status 404 returned error can't find the container with id d336b5999c7868b61dd75e0106e32860e07144c4440eafd7237ff3cb9d1409f4
Jul 24 21:16:11.685089 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.685054    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vkrgq" event={"ID":"37d7e99c-5d52-4c53-a7a2-e46f3910261a","Type":"ContainerStarted","Data":"d336b5999c7868b61dd75e0106e32860e07144c4440eafd7237ff3cb9d1409f4"}
Jul 24 21:16:11.686524 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.686498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl" event={"ID":"cc3f06ac-a6e1-409e-a719-eaf3da600093","Type":"ContainerStarted","Data":"1ad26623874ab5584ccd40337d5789d04428c2803c12b479d609fafafbe507cd"}
Jul 24 21:16:11.686637 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.686545    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl" event={"ID":"cc3f06ac-a6e1-409e-a719-eaf3da600093","Type":"ContainerStarted","Data":"e1d91be2d410676f8fe6e280e8423b9e3565191a557ef9a175ab1ea15b94dc7d"}
Jul 24 21:16:11.686637 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.686585    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:11.713123 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.713072    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl" podStartSLOduration=20.713056313 podStartE2EDuration="20.713056313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:11.711157291 +0000 UTC m=+52.904335246" watchObservedRunningTime="2026-07-24 21:16:11.713056313 +0000 UTC m=+52.906234244"
Jul 24 21:16:11.750116 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.750090    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:11.950681 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:11.950551    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:12.147048 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:12.147004    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:12.358043 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:12.357953    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:12.458382 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:12.458324    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:13.094328 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.094291    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:13.094501 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.094393    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:13.094555 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:13.094530    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:13.094609 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:13.094598    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls podName:70b90561-8565-4a7f-a290-73b7dd0a1f60 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:29.094582637 +0000 UTC m=+70.287760551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xc5zd" (UID: "70b90561-8565-4a7f-a290-73b7dd0a1f60") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:13.094791 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.094772    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70b90561-8565-4a7f-a290-73b7dd0a1f60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:13.353272 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.353202    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:13.552288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.552265    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:13.695296 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.695273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vkrgq" event={"ID":"37d7e99c-5d52-4c53-a7a2-e46f3910261a","Type":"ContainerStarted","Data":"cea7212adbb43cc00456c981dbdbf8dd1cda42da6b5841bd96666749e8d1e720"}
Jul 24 21:16:13.695387 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.695320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vkrgq" event={"ID":"37d7e99c-5d52-4c53-a7a2-e46f3910261a","Type":"ContainerStarted","Data":"10d4af36b342728c12cf799b261734c98963b586ec9724cd61600fd08bad46ec"}
Jul 24 21:16:13.747306 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.747282    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:13.969392 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:13.969316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:14.698356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:14.698328    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:16:14.716520 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:14.716473    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vkrgq" podStartSLOduration=17.608827775 podStartE2EDuration="19.716460675s" totalImagesPullingTime="2.1076329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:14.714863399 +0000 UTC m=+55.908041356" watchObservedRunningTime="2026-07-24 21:16:14.716460675 +0000 UTC m=+55.909638589"
Jul 24 21:16:15.710676 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:15.710631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:15.713200 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:15.713178    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cbefa37b-bdef-4291-8ca3-2271f8f5b02d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qpt4v\" (UID: \"cbefa37b-bdef-4291-8ca3-2271f8f5b02d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:15.806129 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:15.806103    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"
Jul 24 21:16:15.926054 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:15.925983    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v"]
Jul 24 21:16:15.928363 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:15.928331    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbefa37b_bdef_4291_8ca3_2271f8f5b02d.slice/crio-718e1b386a899f9ae9b349bb843e13f9f4e8edddb26b63ce193432115eeea80f WatchSource:0}: Error finding container 718e1b386a899f9ae9b349bb843e13f9f4e8edddb26b63ce193432115eeea80f: Status 404 returned error can't find the container with id 718e1b386a899f9ae9b349bb843e13f9f4e8edddb26b63ce193432115eeea80f
Jul 24 21:16:16.705222 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:16.705190    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v" event={"ID":"cbefa37b-bdef-4291-8ca3-2271f8f5b02d","Type":"ContainerStarted","Data":"718e1b386a899f9ae9b349bb843e13f9f4e8edddb26b63ce193432115eeea80f"}
Jul 24 21:16:17.709531 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:17.709498    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v" event={"ID":"cbefa37b-bdef-4291-8ca3-2271f8f5b02d","Type":"ContainerStarted","Data":"db38c1d357f74c116455ccbd1cc91922274c7a94bda4cefc2153a9e1d31ccd4d"}
Jul 24 21:16:17.739413 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:17.739371    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qpt4v" podStartSLOduration=17.558100959 podStartE2EDuration="18.739357184s" totalImagesPullingTime="1.181256225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:17.739061122 +0000 UTC m=+58.932239059" watchObservedRunningTime="2026-07-24 21:16:17.739357184 +0000 UTC m=+58.932535117"
Jul 24 21:16:19.620395 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:19.620359    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6nm8n"
Jul 24 21:16:20.525156 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:20.525123    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:20.627072 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:20.627042    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.600942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.600911    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"]
Jul 24 21:16:22.603592 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.603572    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.606278 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.606236    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 21:16:22.607552 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.607517    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 21:16:22.607552 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.607537    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 21:16:22.607743 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.607576    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 21:16:22.607743 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.607591    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 21:16:22.607743 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.607517    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 21:16:22.607743 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.607518    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 21:16:22.618485 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.618463    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"]
Jul 24 21:16:22.645975 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.645954    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75c7d9b5f6-55qpl"]
Jul 24 21:16:22.658207 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.658185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-hub\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.658311 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.658216    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.658311 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.658253    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwkx\" (UniqueName: \"kubernetes.io/projected/e86628c5-e96a-4f65-a8a2-f102584f0af8-kube-api-access-cbwkx\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.658382 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.658358    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.658419 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.658393    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-ca\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.658483 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.658467    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86628c5-e96a-4f65-a8a2-f102584f0af8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.669207 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.669189    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.695735 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.695714    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66568f7d4b-gvv94"]
Jul 24 21:16:22.697476 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.697459    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.721811 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.719192    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66568f7d4b-gvv94"]
Jul 24 21:16:22.759028 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5306f000-7fcd-4bf9-8e40-60f294049351-registry-certificates\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759143 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759036    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-bound-sa-token\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759143 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759058    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5306f000-7fcd-4bf9-8e40-60f294049351-installation-pull-secrets\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759143 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759078    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86628c5-e96a-4f65-a8a2-f102584f0af8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.759320 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759134    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5306f000-7fcd-4bf9-8e40-60f294049351-trusted-ca\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759320 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759172    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-registry-tls\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759320 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759199    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5306f000-7fcd-4bf9-8e40-60f294049351-ca-trust-extracted\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759320 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759250    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-hub\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.759320 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759282    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.759534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759344    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwkx\" (UniqueName: \"kubernetes.io/projected/e86628c5-e96a-4f65-a8a2-f102584f0af8-kube-api-access-cbwkx\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.759534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759371    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5306f000-7fcd-4bf9-8e40-60f294049351-image-registry-private-configuration\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759517    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.759694 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759555    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-ca\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.759694 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn576\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-kube-api-access-xn576\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.759809 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.759768    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86628c5-e96a-4f65-a8a2-f102584f0af8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.762082 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.762063    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.762209 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.762184    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-hub\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.762301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.762109    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-ca\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.762301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.762109    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e86628c5-e96a-4f65-a8a2-f102584f0af8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.762301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.762275    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.768261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.768243    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwkx\" (UniqueName: \"kubernetes.io/projected/e86628c5-e96a-4f65-a8a2-f102584f0af8-kube-api-access-cbwkx\") pod \"cluster-proxy-proxy-agent-7f5689d747-ksk95\" (UID: \"e86628c5-e96a-4f65-a8a2-f102584f0af8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.860617 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860554    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5306f000-7fcd-4bf9-8e40-60f294049351-registry-certificates\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860617 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860585    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-bound-sa-token\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860617 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860607    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5306f000-7fcd-4bf9-8e40-60f294049351-installation-pull-secrets\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860827 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5306f000-7fcd-4bf9-8e40-60f294049351-trusted-ca\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860827 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860770    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-registry-tls\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860827 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5306f000-7fcd-4bf9-8e40-60f294049351-ca-trust-extracted\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860972 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860886    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5306f000-7fcd-4bf9-8e40-60f294049351-image-registry-private-configuration\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.860972 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.860924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn576\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-kube-api-access-xn576\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.861269 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.861223    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5306f000-7fcd-4bf9-8e40-60f294049351-ca-trust-extracted\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.861534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.861501    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5306f000-7fcd-4bf9-8e40-60f294049351-registry-certificates\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.861616 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.861569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5306f000-7fcd-4bf9-8e40-60f294049351-trusted-ca\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.863074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.863053    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5306f000-7fcd-4bf9-8e40-60f294049351-installation-pull-secrets\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.863233 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.863214    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-registry-tls\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.863373 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.863356    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5306f000-7fcd-4bf9-8e40-60f294049351-image-registry-private-configuration\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.868395 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.868369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.881403 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.881373    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-bound-sa-token\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.881568 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.881549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn576\" (UniqueName: \"kubernetes.io/projected/5306f000-7fcd-4bf9-8e40-60f294049351-kube-api-access-xn576\") pod \"image-registry-66568f7d4b-gvv94\" (UID: \"5306f000-7fcd-4bf9-8e40-60f294049351\") " pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:22.921800 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.921778    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"
Jul 24 21:16:22.969517 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:22.969481    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:23.005130 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.005101    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:23.051394 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.050892    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95"]
Jul 24 21:16:23.057090 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:23.057061    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode86628c5_e96a_4f65_a8a2_f102584f0af8.slice/crio-cea540b8212c28621f1d3d497b413111d58e2a16d5020ae3dd88edaa8885f230 WatchSource:0}: Error finding container cea540b8212c28621f1d3d497b413111d58e2a16d5020ae3dd88edaa8885f230: Status 404 returned error can't find the container with id cea540b8212c28621f1d3d497b413111d58e2a16d5020ae3dd88edaa8885f230
Jul 24 21:16:23.057588 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.057559    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:23.157102 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:23.157081    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5306f000_7fcd_4bf9_8e40_60f294049351.slice/crio-d25ad590a2adcd7f6e4efc24225eff3176501f6e64db8aae3b5c994ae04601ba WatchSource:0}: Error finding container d25ad590a2adcd7f6e4efc24225eff3176501f6e64db8aae3b5c994ae04601ba: Status 404 returned error can't find the container with id d25ad590a2adcd7f6e4efc24225eff3176501f6e64db8aae3b5c994ae04601ba
Jul 24 21:16:23.162496 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.162476    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66568f7d4b-gvv94"]
Jul 24 21:16:23.730092 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.730049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94" event={"ID":"5306f000-7fcd-4bf9-8e40-60f294049351","Type":"ContainerStarted","Data":"2f7506ea52d54c09eb6d2a9633c0bb8b57f7730fb4cae4a370eebfb62d2b0bc0"}
Jul 24 21:16:23.730551 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.730113    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94" event={"ID":"5306f000-7fcd-4bf9-8e40-60f294049351","Type":"ContainerStarted","Data":"d25ad590a2adcd7f6e4efc24225eff3176501f6e64db8aae3b5c994ae04601ba"}
Jul 24 21:16:23.730551 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.730166    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:23.731133 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.731112    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95" event={"ID":"e86628c5-e96a-4f65-a8a2-f102584f0af8","Type":"ContainerStarted","Data":"cea540b8212c28621f1d3d497b413111d58e2a16d5020ae3dd88edaa8885f230"}
Jul 24 21:16:23.775098 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:23.775049    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94" podStartSLOduration=1.775034195 podStartE2EDuration="1.775034195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:23.774795499 +0000 UTC m=+64.967973471" watchObservedRunningTime="2026-07-24 21:16:23.775034195 +0000 UTC m=+64.968212133"
Jul 24 21:16:24.703399 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:24.703366    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vkrgq"
Jul 24 21:16:25.082002 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.081911    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:16:25.082344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.082020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:16:25.084964 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.084939    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:16:25.084964 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.084946    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:16:25.095548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.095526    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694177d1-b8fa-43ec-9ad0-dbd7aa96111f-metrics-certs\") pod \"network-metrics-daemon-5sl4p\" (UID: \"694177d1-b8fa-43ec-9ad0-dbd7aa96111f\") " pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:16:25.097214 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.097189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf-original-pull-secret\") pod \"global-pull-secret-syncer-7kk4m\" (UID: \"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf\") " pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:16:25.136912 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.136788    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7jm8m\""
Jul 24 21:16:25.140363 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.140337    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kk4m"
Jul 24 21:16:25.144574 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.144552    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5sl4p"
Jul 24 21:16:25.182731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.182706    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:16:25.186405 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.186373    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:16:25.197156 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.197132    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:16:25.207072 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.207023    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbptj\" (UniqueName: \"kubernetes.io/projected/2dae7df5-d719-4b5b-83c5-be9c6af6b24c-kube-api-access-qbptj\") pod \"network-check-target-n6hjf\" (UID: \"2dae7df5-d719-4b5b-83c5-be9c6af6b24c\") " pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:16:25.417993 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.417956    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7kk4m"]
Jul 24 21:16:25.421327 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:25.421298    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67cabf5a_03ac_492a_9bc0_79d3c1d8dcdf.slice/crio-f7b75073c11bb00c838ab48a28f79246b49e32fbcd23bbe21641c621dc9feb2a WatchSource:0}: Error finding container f7b75073c11bb00c838ab48a28f79246b49e32fbcd23bbe21641c621dc9feb2a: Status 404 returned error can't find the container with id f7b75073c11bb00c838ab48a28f79246b49e32fbcd23bbe21641c621dc9feb2a
Jul 24 21:16:25.436201 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.436135    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5sl4p"]
Jul 24 21:16:25.438741 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:25.438706    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694177d1_b8fa_43ec_9ad0_dbd7aa96111f.slice/crio-ce21031a727153b32b3fffde0f0d91aeca6169bb771f38bd6384cda2ea62e049 WatchSource:0}: Error finding container ce21031a727153b32b3fffde0f0d91aeca6169bb771f38bd6384cda2ea62e049: Status 404 returned error can't find the container with id ce21031a727153b32b3fffde0f0d91aeca6169bb771f38bd6384cda2ea62e049
Jul 24 21:16:25.449943 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.449923    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66lqr\""
Jul 24 21:16:25.457500 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.457482    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:16:25.742470 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.742429    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5sl4p" event={"ID":"694177d1-b8fa-43ec-9ad0-dbd7aa96111f","Type":"ContainerStarted","Data":"ce21031a727153b32b3fffde0f0d91aeca6169bb771f38bd6384cda2ea62e049"}
Jul 24 21:16:25.743697 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:25.743647    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7kk4m" event={"ID":"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf","Type":"ContainerStarted","Data":"f7b75073c11bb00c838ab48a28f79246b49e32fbcd23bbe21641c621dc9feb2a"}
Jul 24 21:16:26.691885 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:26.691803    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n6hjf"]
Jul 24 21:16:26.764456 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:26.764429    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dae7df5_d719_4b5b_83c5_be9c6af6b24c.slice/crio-86027fa4a0777f158958beab322d03bfeda11ac1051031bac5530f71eec24452 WatchSource:0}: Error finding container 86027fa4a0777f158958beab322d03bfeda11ac1051031bac5530f71eec24452: Status 404 returned error can't find the container with id 86027fa4a0777f158958beab322d03bfeda11ac1051031bac5530f71eec24452
Jul 24 21:16:27.563621 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.563572    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.629029 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.628853    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.695477 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.695442    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.753053 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.752987    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6hjf" event={"ID":"2dae7df5-d719-4b5b-83c5-be9c6af6b24c","Type":"ContainerStarted","Data":"86027fa4a0777f158958beab322d03bfeda11ac1051031bac5530f71eec24452"}
Jul 24 21:16:27.755811 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.755781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95" event={"ID":"e86628c5-e96a-4f65-a8a2-f102584f0af8","Type":"ContainerStarted","Data":"c03a96f8bb16de50711ef4ee2a0c3352a8a535bb5c2a2b3b321e62f0a998dea1"}
Jul 24 21:16:27.755936 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.755825    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95" event={"ID":"e86628c5-e96a-4f65-a8a2-f102584f0af8","Type":"ContainerStarted","Data":"f2f85c0baaf8ba295c461155d7303f0319b578d9881f930347ab743a53744875"}
Jul 24 21:16:27.755936 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.755849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95" event={"ID":"e86628c5-e96a-4f65-a8a2-f102584f0af8","Type":"ContainerStarted","Data":"40e3099479418f5453f70eddcc4d0599912e81c2a9975456a355acc8a485acc7"}
Jul 24 21:16:27.776821 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:27.776771    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5689d747-ksk95" podStartSLOduration=2.025742061 podStartE2EDuration="5.776750301s" totalImagesPullingTime="3.75100824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:27.776405749 +0000 UTC m=+68.969583686" watchObservedRunningTime="2026-07-24 21:16:27.776750301 +0000 UTC m=+68.969928243"
Jul 24 21:16:28.762236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:28.762129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5sl4p" event={"ID":"694177d1-b8fa-43ec-9ad0-dbd7aa96111f","Type":"ContainerStarted","Data":"e55747564f2be263cb21194e99730e412120e5ff4dc12103b3301d4b5d12ae4d"}
Jul 24 21:16:28.762236 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:28.762195    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5sl4p" event={"ID":"694177d1-b8fa-43ec-9ad0-dbd7aa96111f","Type":"ContainerStarted","Data":"dc7014e50eb1d453f1df601fb998199acf008fd8c82be8a4eb51e45364465f06"}
Jul 24 21:16:28.781867 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:28.781645    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5sl4p" podStartSLOduration=66.399219013 podStartE2EDuration="1m8.781629604s" totalImagesPullingTime="2.382410591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:28.779100826 +0000 UTC m=+69.972278790" watchObservedRunningTime="2026-07-24 21:16:28.781629604 +0000 UTC m=+69.974807541"
Jul 24 21:16:28.870100 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:28.870048    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.974889 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:28.974849    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:29.073641 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.073542    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:29.114890 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.114855    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:29.117565 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.117541    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70b90561-8565-4a7f-a290-73b7dd0a1f60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xc5zd\" (UID: \"70b90561-8565-4a7f-a290-73b7dd0a1f60\") " pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:29.177135 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.177104    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:29.351724 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.351625    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cfdvq\""
Jul 24 21:16:29.360063 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.359709    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xc5zd"
Jul 24 21:16:29.464147 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:29.464114    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:31.481095 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.481072    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xc5zd"]
Jul 24 21:16:31.483440 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:31.483410    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b90561_8565_4a7f_a290_73b7dd0a1f60.slice/crio-8bd67f59429db5df7e6638bbdf8c4c3f2f67b9fb1fcd33858430063015af637f WatchSource:0}: Error finding container 8bd67f59429db5df7e6638bbdf8c4c3f2f67b9fb1fcd33858430063015af637f: Status 404 returned error can't find the container with id 8bd67f59429db5df7e6638bbdf8c4c3f2f67b9fb1fcd33858430063015af637f
Jul 24 21:16:31.773454 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.773419    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7kk4m" event={"ID":"67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf","Type":"ContainerStarted","Data":"2decf7acfac2ecbca8d74e09898b8b28e98e98cbaaa3058be0dacc19e4851f3b"}
Jul 24 21:16:31.774853 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.774830    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n6hjf" event={"ID":"2dae7df5-d719-4b5b-83c5-be9c6af6b24c","Type":"ContainerStarted","Data":"488586a1b8f59c4bf7da07f5925c24a8626167ab7e86ec385a9ac4e363281cfc"}
Jul 24 21:16:31.774991 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.774876    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:16:31.775963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.775938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xc5zd" event={"ID":"70b90561-8565-4a7f-a290-73b7dd0a1f60","Type":"ContainerStarted","Data":"8bd67f59429db5df7e6638bbdf8c4c3f2f67b9fb1fcd33858430063015af637f"}
Jul 24 21:16:31.825938 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.825897    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7kk4m" podStartSLOduration=65.885176398 podStartE2EDuration="1m11.82588546s" totalImagesPullingTime="5.940709062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:31.79839428 +0000 UTC m=+72.991572218" watchObservedRunningTime="2026-07-24 21:16:31.82588546 +0000 UTC m=+73.019063396"
Jul 24 21:16:31.826220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.826197    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n6hjf" podStartSLOduration=67.268800459 podStartE2EDuration="1m11.826192388s" totalImagesPullingTime="4.557391929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:31.825242078 +0000 UTC m=+73.018420014" watchObservedRunningTime="2026-07-24 21:16:31.826192388 +0000 UTC m=+73.019370325"
Jul 24 21:16:31.859914 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.859880    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:31.924195 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:31.924162    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:32.653581 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:32.653554    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:33.869062 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:33.869031    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:33.969908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:33.969874    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.206927 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:34.206837    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.318152 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:34.318120    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.418894 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:34.418860    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.694579 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:34.694544    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.766228 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:34.766190    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:36.796291 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:36.796254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xc5zd" event={"ID":"70b90561-8565-4a7f-a290-73b7dd0a1f60","Type":"ContainerStarted","Data":"e218291c8168c993aee400ac31daeb5d06bfa02d9709a17f5f22193ed562b195"}
Jul 24 21:16:38.355183 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:38.355146    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:38.448988 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:38.448957    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:38.637978 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:38.637933    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:38.789621 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:38.789589    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:38.894261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:38.894170    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.014039 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:39.014006    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.091214 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:39.091183    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.195940 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:39.195874    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.260947 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:39.260918    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.401616 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:39.401583    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.810323 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:39.810217    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.075565 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:40.075506    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.237038 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:40.236988    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.618447 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:40.618415    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.812903 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:40.812859    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xc5zd" event={"ID":"70b90561-8565-4a7f-a290-73b7dd0a1f60","Type":"ContainerStarted","Data":"14639d99f1507cb1b98ed8e070f119826eb62a09236787c9f98c3416458d0888"}
Jul 24 21:16:40.947926 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:40.947840    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:41.045575 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:41.045538    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:41.216141 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:41.216068    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:41.316942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:41.316896    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.083137 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.083105    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.184522 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.184498    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.243280 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.243246    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.338561 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.338475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.480092 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.480065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.581262 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.581230    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.616791 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.616727    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-26llm"]
Jul 24 21:16:42.632510 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.631517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.635202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.635170    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 21:16:42.635202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.635202    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 21:16:42.635365 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.635202    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 21:16:42.635365 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.635173    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 21:16:42.636155 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.636130    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 21:16:42.636261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.636186    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6n2pd\""
Jul 24 21:16:42.636331 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.636130    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 21:16:42.679809 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.679719    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726785    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-accelerators-collector-config\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726854    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-sys\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726894    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-tls\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726925    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-textfile\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726971    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-wtmp\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.726999    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/227c9960-a563-481c-999d-b8eeb54abe48-metrics-client-ca\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.727021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-root\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.727185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.727060    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcb2\" (UniqueName: \"kubernetes.io/projected/227c9960-a563-481c-999d-b8eeb54abe48-kube-api-access-rvcb2\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.775221 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.775182    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.827411 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-wtmp\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827411 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/227c9960-a563-481c-999d-b8eeb54abe48-metrics-client-ca\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827439    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-root\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827478    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcb2\" (UniqueName: \"kubernetes.io/projected/227c9960-a563-481c-999d-b8eeb54abe48-kube-api-access-rvcb2\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-accelerators-collector-config\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827551    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827575    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-sys\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827613    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-tls\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.827926 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.827647    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-textfile\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.828119    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-textfile\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.828388    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-wtmp\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.828601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-accelerators-collector-config\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.828892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/227c9960-a563-481c-999d-b8eeb54abe48-metrics-client-ca\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.828952    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-root\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.828993    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/227c9960-a563-481c-999d-b8eeb54abe48-sys\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:42.829067    2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 21:16:42.829273 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:16:42.829120    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-tls podName:227c9960-a563-481c-999d-b8eeb54abe48 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:43.329106762 +0000 UTC m=+84.522284677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-tls") pod "node-exporter-26llm" (UID: "227c9960-a563-481c-999d-b8eeb54abe48") : secret "node-exporter-tls" not found
Jul 24 21:16:42.839444 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.839004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.844697 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.844637    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcb2\" (UniqueName: \"kubernetes.io/projected/227c9960-a563-481c-999d-b8eeb54abe48-kube-api-access-rvcb2\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:42.874050 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.874030    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:42.973778 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:42.973717    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:43.178678 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.178583    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:43.279165 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.279134    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:43.331819 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.331793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-tls\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:43.334441 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.334411    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/227c9960-a563-481c-999d-b8eeb54abe48-node-exporter-tls\") pod \"node-exporter-26llm\" (UID: \"227c9960-a563-481c-999d-b8eeb54abe48\") " pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:43.380015 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.379975    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:43.544767 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.544703    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-26llm"
Jul 24 21:16:43.606382 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.606354    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:43.673194 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:16:43.673168    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227c9960_a563_481c_999d_b8eeb54abe48.slice/crio-4a1c2c989af560ae353c8056c2a4860fec70e3bf0115964d99b2816e05ea74b4 WatchSource:0}: Error finding container 4a1c2c989af560ae353c8056c2a4860fec70e3bf0115964d99b2816e05ea74b4: Status 404 returned error can't find the container with id 4a1c2c989af560ae353c8056c2a4860fec70e3bf0115964d99b2816e05ea74b4
Jul 24 21:16:43.767639 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.767611    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:43.824917 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.824857    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26llm" event={"ID":"227c9960-a563-481c-999d-b8eeb54abe48","Type":"ContainerStarted","Data":"4a1c2c989af560ae353c8056c2a4860fec70e3bf0115964d99b2816e05ea74b4"}
Jul 24 21:16:43.827025 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:43.826981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xc5zd" event={"ID":"70b90561-8565-4a7f-a290-73b7dd0a1f60","Type":"ContainerStarted","Data":"f2fd8557cecbcff4b10ba42e10f7b90a9f6e4a358caa21e64cdd60795d4a2ef7"}
Jul 24 21:16:44.146906 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:44.146881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:44.739482 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:44.739457    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66568f7d4b-gvv94"
Jul 24 21:16:44.770585 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:44.770525    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xc5zd" podStartSLOduration=35.655519823 podStartE2EDuration="47.770506427s" totalImagesPullingTime="12.114986604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:43.853360359 +0000 UTC m=+85.046538297" watchObservedRunningTime="2026-07-24 21:16:44.770506427 +0000 UTC m=+85.963684367"
Jul 24 21:16:44.832246 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:44.832213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26llm" event={"ID":"227c9960-a563-481c-999d-b8eeb54abe48","Type":"ContainerStarted","Data":"12c7f96b5c5bf5afbeb4b6c82a63299c636c460e9a8e43066c2c96cb72c0066d"}
Jul 24 21:16:44.901056 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:44.901028    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:44.982399 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:44.982325    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:45.611051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:45.611023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:45.676213 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:45.676178    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:45.744821 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:45.744788    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:45.837313 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:45.837284    2585 generic.go:356] "Generic (PLEG): container finished" podID="227c9960-a563-481c-999d-b8eeb54abe48" containerID="12c7f96b5c5bf5afbeb4b6c82a63299c636c460e9a8e43066c2c96cb72c0066d" exitCode=0
Jul 24 21:16:45.837458 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:45.837320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26llm" event={"ID":"227c9960-a563-481c-999d-b8eeb54abe48","Type":"ContainerDied","Data":"12c7f96b5c5bf5afbeb4b6c82a63299c636c460e9a8e43066c2c96cb72c0066d"}
Jul 24 21:16:45.845001 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:45.844979    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:46.497791 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:46.497763    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:46.609921 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:46.609884    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:46.843701 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:46.843609    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26llm" event={"ID":"227c9960-a563-481c-999d-b8eeb54abe48","Type":"ContainerStarted","Data":"80b2c9b4932812aee4dd91b9da1c4d55f792ad5ca14197f12fbb17bc86a8b437"}
Jul 24 21:16:46.843701 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:46.843659    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-26llm" event={"ID":"227c9960-a563-481c-999d-b8eeb54abe48","Type":"ContainerStarted","Data":"aee9e765a4b9dbd2c468edff27f2fb7ee24133d65131381142ff69dc36c7c849"}
Jul 24 21:16:46.865042 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:46.865000    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-26llm" podStartSLOduration=3.772773254 podStartE2EDuration="4.864985711s" totalImagesPullingTime="1.016765453s" totalInitContainerRuntime="75.447004ms" isStatefulPod=true podCreationTimestamp="2026-07-24 21:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:46.864632665 +0000 UTC m=+88.057810601" watchObservedRunningTime="2026-07-24 21:16:46.864985711 +0000 UTC m=+88.058163704"
Jul 24 21:16:47.365350 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.365315    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:47.424433 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.424390    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:47.554642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.554610    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:47.664356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.664322    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl" podUID="cc3f06ac-a6e1-409e-a719-eaf3da600093" containerName="registry" containerID="cri-o://1ad26623874ab5584ccd40337d5789d04428c2803c12b479d609fafafbe507cd" gracePeriod=30
Jul 24 21:16:47.849761 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.849729    2585 generic.go:356] "Generic (PLEG): container finished" podID="cc3f06ac-a6e1-409e-a719-eaf3da600093" containerID="1ad26623874ab5584ccd40337d5789d04428c2803c12b479d609fafafbe507cd" exitCode=0
Jul 24 21:16:47.850083 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.849800    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl" event={"ID":"cc3f06ac-a6e1-409e-a719-eaf3da600093","Type":"ContainerDied","Data":"1ad26623874ab5584ccd40337d5789d04428c2803c12b479d609fafafbe507cd"}
Jul 24 21:16:47.905814 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.905794    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:47.967901 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.967839    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.967901 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.967875    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-installation-pull-secrets\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.967924    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8vzd\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-kube-api-access-h8vzd\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.967963    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-bound-sa-token\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.967997    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-certificates\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.968019    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc3f06ac-a6e1-409e-a719-eaf3da600093-ca-trust-extracted\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.968056    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-trusted-ca\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968313 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.968088    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-image-registry-private-configuration\") pod \"cc3f06ac-a6e1-409e-a719-eaf3da600093\" (UID: \"cc3f06ac-a6e1-409e-a719-eaf3da600093\") "
Jul 24 21:16:47.968814 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.968700    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:16:47.969196 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.969171    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:16:47.970837 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.970810    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:16:47.970961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.970817    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:16:47.970961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.970925    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:16:47.971070 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.970954    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:16:47.971159 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.971142    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-kube-api-access-h8vzd" (OuterVolumeSpecName: "kube-api-access-h8vzd") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "kube-api-access-h8vzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:16:47.977072 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:47.977048    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc3f06ac-a6e1-409e-a719-eaf3da600093-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cc3f06ac-a6e1-409e-a719-eaf3da600093" (UID: "cc3f06ac-a6e1-409e-a719-eaf3da600093"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:16:48.068626 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068592    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-trusted-ca\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068626 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068624    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-image-registry-private-configuration\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068626 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068634    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068804 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068644    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc3f06ac-a6e1-409e-a719-eaf3da600093-installation-pull-secrets\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068804 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068653    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8vzd\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-kube-api-access-h8vzd\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068804 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068677    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc3f06ac-a6e1-409e-a719-eaf3da600093-bound-sa-token\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068804 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068686    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc3f06ac-a6e1-409e-a719-eaf3da600093-registry-certificates\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.068804 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.068694    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc3f06ac-a6e1-409e-a719-eaf3da600093-ca-trust-extracted\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:16:48.854079 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.854042    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl" event={"ID":"cc3f06ac-a6e1-409e-a719-eaf3da600093","Type":"ContainerDied","Data":"e1d91be2d410676f8fe6e280e8423b9e3565191a557ef9a175ab1ea15b94dc7d"}
Jul 24 21:16:48.854079 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.854076    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c7d9b5f6-55qpl"
Jul 24 21:16:48.854528 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.854097    2585 scope.go:117] "RemoveContainer" containerID="1ad26623874ab5584ccd40337d5789d04428c2803c12b479d609fafafbe507cd"
Jul 24 21:16:48.876383 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.876360    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75c7d9b5f6-55qpl"]
Jul 24 21:16:48.884861 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.884842    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75c7d9b5f6-55qpl"]
Jul 24 21:16:48.919606 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:48.919578    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:49.083949 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:49.083924    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:49.325999 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:49.325959    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc3f06ac-a6e1-409e-a719-eaf3da600093" path="/var/lib/kubelet/pods/cc3f06ac-a6e1-409e-a719-eaf3da600093/volumes"
Jul 24 21:16:50.491190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:50.491153    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.555440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:50.555412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.878946 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:51.878897    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.979711 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:51.979636    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:53.139443 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:53.139410    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:53.201523 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:53.201500    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.033872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:54.033841    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.144796 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:54.144746    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.244263 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:54.244235    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.339460 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:54.339367    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:58.424892 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:58.424857    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:58.486467 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:16:58.486439    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:02.783843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:02.783807    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n6hjf"
Jul 24 21:17:04.229490 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:04.229453    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:04.294491 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:04.294458    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.495808 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:05.495762    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.557956 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:05.557919    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:20.492517 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:20.492478    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:20.593091 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:17:20.593056    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:01.871995 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:01.871962    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:02.080481 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:02.080440    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:02.181183 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:02.181100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:11.050283 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:11.050195    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:11.150634 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:11.150607    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:21.478814 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:21.478779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:21.541567 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:21.541535    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:21.597025 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:21.596978    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:36.804941 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:36.804889    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:37.880793 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:37.880719    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:37.937060 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:18:37.937015    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:17.119147 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:17.119080    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:17.207098 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:17.207030    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:29.228628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:29.228593    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:29.284325 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:29.284298    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:31.743383 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:31.743338    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:31.844204 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:31.844171    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:35.500515 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:35.500480    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:35.602786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:35.602752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:42.756057 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:42.755983    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:42.855454 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:42.855412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:42.923780 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:19:42.923752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:19.177989 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:19.177962    2585 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 21:20:36.729411 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:36.729378    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:37.956147 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:37.956111    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:38.310227 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.310151    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"]
Jul 24 21:20:38.310437 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.310423    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc3f06ac-a6e1-409e-a719-eaf3da600093" containerName="registry"
Jul 24 21:20:38.310476 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.310441    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc3f06ac-a6e1-409e-a719-eaf3da600093" containerName="registry"
Jul 24 21:20:38.310509 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.310498    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc3f06ac-a6e1-409e-a719-eaf3da600093" containerName="registry"
Jul 24 21:20:38.312396 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.312377    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.315830 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.315809    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-45jvb\""
Jul 24 21:20:38.316297 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.316277    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 24 21:20:38.317095 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.317081    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 24 21:20:38.326951 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.326926    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"]
Jul 24 21:20:38.396220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.396195    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.396319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.396225    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvll\" (UniqueName: \"kubernetes.io/projected/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-kube-api-access-wcvll\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.396319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.396304    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.497383 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.497355    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.497491 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.497394    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.497491 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.497414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvll\" (UniqueName: \"kubernetes.io/projected/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-kube-api-access-wcvll\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.497721 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.497702    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.497796 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.497777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.507867 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.507837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvll\" (UniqueName: \"kubernetes.io/projected/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-kube-api-access-wcvll\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.621247 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.621225    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:38.751638 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.751589    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"]
Jul 24 21:20:38.753714 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:20:38.753688    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1bd8b8_b52d_4bd8_b504_eb9369febe32.slice/crio-09fa98779f9388118c1572fd39e036936f27ca48f06d1dca2a21d7f0e9d4859e WatchSource:0}: Error finding container 09fa98779f9388118c1572fd39e036936f27ca48f06d1dca2a21d7f0e9d4859e: Status 404 returned error can't find the container with id 09fa98779f9388118c1572fd39e036936f27ca48f06d1dca2a21d7f0e9d4859e
Jul 24 21:20:38.755601 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:38.755585    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:20:39.599393 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:39.599360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn" event={"ID":"dc1bd8b8-b52d-4bd8-b504-eb9369febe32","Type":"ContainerStarted","Data":"09fa98779f9388118c1572fd39e036936f27ca48f06d1dca2a21d7f0e9d4859e"}
Jul 24 21:20:45.623024 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:45.622985    2585 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerID="448a37f481bd5accaa10f5f7e9cbdffd2fe291c3258a71e8cdcab9c0c3b755df" exitCode=0
Jul 24 21:20:45.623373 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:45.623057    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn" event={"ID":"dc1bd8b8-b52d-4bd8-b504-eb9369febe32","Type":"ContainerDied","Data":"448a37f481bd5accaa10f5f7e9cbdffd2fe291c3258a71e8cdcab9c0c3b755df"}
Jul 24 21:20:47.631343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:47.631308    2585 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerID="3018b589263e7d0ff200ea8cd1604caf1321430caeed65515a539383958b4046" exitCode=0
Jul 24 21:20:47.631645 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:47.631388    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn" event={"ID":"dc1bd8b8-b52d-4bd8-b504-eb9369febe32","Type":"ContainerDied","Data":"3018b589263e7d0ff200ea8cd1604caf1321430caeed65515a539383958b4046"}
Jul 24 21:20:54.659103 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:54.659067    2585 generic.go:356] "Generic (PLEG): container finished" podID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerID="a7051707f131c1c852e534a2e532c514dcda71f91cb3f79d70a4337aac8b1a5c" exitCode=0
Jul 24 21:20:54.659468 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:54.659109    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn" event={"ID":"dc1bd8b8-b52d-4bd8-b504-eb9369febe32","Type":"ContainerDied","Data":"a7051707f131c1c852e534a2e532c514dcda71f91cb3f79d70a4337aac8b1a5c"}
Jul 24 21:20:55.784653 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.784631    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:55.933765 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.933652    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvll\" (UniqueName: \"kubernetes.io/projected/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-kube-api-access-wcvll\") pod \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") "
Jul 24 21:20:55.933765 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.933752    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-bundle\") pod \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") "
Jul 24 21:20:55.933994 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.933775    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-util\") pod \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\" (UID: \"dc1bd8b8-b52d-4bd8-b504-eb9369febe32\") "
Jul 24 21:20:55.934356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.934331    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-bundle" (OuterVolumeSpecName: "bundle") pod "dc1bd8b8-b52d-4bd8-b504-eb9369febe32" (UID: "dc1bd8b8-b52d-4bd8-b504-eb9369febe32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:20:55.936023 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.935989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-kube-api-access-wcvll" (OuterVolumeSpecName: "kube-api-access-wcvll") pod "dc1bd8b8-b52d-4bd8-b504-eb9369febe32" (UID: "dc1bd8b8-b52d-4bd8-b504-eb9369febe32"). InnerVolumeSpecName "kube-api-access-wcvll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:20:55.937967 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:55.937948    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-util" (OuterVolumeSpecName: "util") pod "dc1bd8b8-b52d-4bd8-b504-eb9369febe32" (UID: "dc1bd8b8-b52d-4bd8-b504-eb9369febe32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:20:56.034963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:56.034935    2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-bundle\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:20:56.034963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:56.034959    2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-util\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:20:56.034963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:56.034967    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvll\" (UniqueName: \"kubernetes.io/projected/dc1bd8b8-b52d-4bd8-b504-eb9369febe32-kube-api-access-wcvll\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:20:56.667722 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:56.667692    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn"
Jul 24 21:20:56.667722 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:56.667699    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5sj4rn" event={"ID":"dc1bd8b8-b52d-4bd8-b504-eb9369febe32","Type":"ContainerDied","Data":"09fa98779f9388118c1572fd39e036936f27ca48f06d1dca2a21d7f0e9d4859e"}
Jul 24 21:20:56.667929 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:56.667743    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09fa98779f9388118c1572fd39e036936f27ca48f06d1dca2a21d7f0e9d4859e"
Jul 24 21:20:59.492418 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:59.492381    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:59.593292 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:20:59.593259    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:04.995348 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:04.995314    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:05.269639 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269558    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-4bb8x"]
Jul 24 21:21:05.269902 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269885    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="pull"
Jul 24 21:21:05.269980 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269911    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="pull"
Jul 24 21:21:05.269980 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269926    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="util"
Jul 24 21:21:05.269980 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269934    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="util"
Jul 24 21:21:05.269980 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269950    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="extract"
Jul 24 21:21:05.269980 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.269958    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="extract"
Jul 24 21:21:05.270214 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.270028    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1bd8b8-b52d-4bd8-b504-eb9369febe32" containerName="extract"
Jul 24 21:21:05.273230 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.273210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.279240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.276555    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 24 21:21:05.279240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.276562    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 24 21:21:05.279240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.276908    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 24 21:21:05.279240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.277958    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 24 21:21:05.279240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.278196    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6sg4v\""
Jul 24 21:21:05.279240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.278430    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 24 21:21:05.282319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.282061    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:05.283597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.283550    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-4bb8x"]
Jul 24 21:21:05.397969 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.397938    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jx9c\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-kube-api-access-8jx9c\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.398104 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.397981    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-certificates\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.398104 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.398052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/876f6180-7cf5-4176-813b-af69e63f3f2e-cabundle0\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.474417 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.474381    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"]
Jul 24 21:21:05.477699 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.477678    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.479629 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.479594    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:05.480476 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.480459    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 24 21:21:05.489494 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.489474    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"]
Jul 24 21:21:05.498609 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.498588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/876f6180-7cf5-4176-813b-af69e63f3f2e-cabundle0\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.498815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.498792    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jx9c\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-kube-api-access-8jx9c\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.498977 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.498950    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-certificates\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.499092 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.499072    2585 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 24 21:21:05.499163 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.499100    2585 secret.go:281] references non-existent secret key: ca.crt
Jul 24 21:21:05.499163 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.499111    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 24 21:21:05.499163 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.499128    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-4bb8x: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 24 21:21:05.499300 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.499190    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-certificates podName:876f6180-7cf5-4176-813b-af69e63f3f2e nodeName:}" failed. No retries permitted until 2026-07-24 21:21:05.999170033 +0000 UTC m=+347.192347973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-certificates") pod "keda-operator-77f648d694-4bb8x" (UID: "876f6180-7cf5-4176-813b-af69e63f3f2e") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 24 21:21:05.499361 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.499310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/876f6180-7cf5-4176-813b-af69e63f3f2e-cabundle0\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.512371 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.512345    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jx9c\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-kube-api-access-8jx9c\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:05.599645 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.599577    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr5vw\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-kube-api-access-lr5vw\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.599645 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.599631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f25abda8-754d-48a7-85b3-6151343db03d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.599838 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.599657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-certificates\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.604825 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.604793    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:05.700034 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.700005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr5vw\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-kube-api-access-lr5vw\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.700190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.700059    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f25abda8-754d-48a7-85b3-6151343db03d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.700190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.700081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-certificates\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.700305 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.700215    2585 secret.go:281] references non-existent secret key: tls.crt
Jul 24 21:21:05.700305 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.700230    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 24 21:21:05.700305 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.700251    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss: references non-existent secret key: tls.crt
Jul 24 21:21:05.700422 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:21:05.700314    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-certificates podName:f25abda8-754d-48a7-85b3-6151343db03d nodeName:}" failed. No retries permitted until 2026-07-24 21:21:06.200293876 +0000 UTC m=+347.393471808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-certificates") pod "keda-metrics-apiserver-55557df8c-tq4ss" (UID: "f25abda8-754d-48a7-85b3-6151343db03d") : references non-existent secret key: tls.crt
Jul 24 21:21:05.700503 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.700484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f25abda8-754d-48a7-85b3-6151343db03d-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.709608 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.709574    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr5vw\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-kube-api-access-lr5vw\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:05.741519 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.741488    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:05.877784 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.877744    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:05.975519 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:05.975493    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:06.003051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.003025    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-certificates\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:06.005599 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.005575    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/876f6180-7cf5-4176-813b-af69e63f3f2e-certificates\") pod \"keda-operator-77f648d694-4bb8x\" (UID: \"876f6180-7cf5-4176-813b-af69e63f3f2e\") " pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:06.077110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.077065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:06.187208 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.187151    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:06.205548 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.205526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-certificates\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:06.208286 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.208265    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f25abda8-754d-48a7-85b3-6151343db03d-certificates\") pod \"keda-metrics-apiserver-55557df8c-tq4ss\" (UID: \"f25abda8-754d-48a7-85b3-6151343db03d\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:06.316300 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.316171    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-4bb8x"]
Jul 24 21:21:06.318503 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:21:06.318475    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876f6180_7cf5_4176_813b_af69e63f3f2e.slice/crio-a3e48d217dd854d0aac6a11dfd2c58a891ce2fbde7e95e5e4b476fd5356d3227 WatchSource:0}: Error finding container a3e48d217dd854d0aac6a11dfd2c58a891ce2fbde7e95e5e4b476fd5356d3227: Status 404 returned error can't find the container with id a3e48d217dd854d0aac6a11dfd2c58a891ce2fbde7e95e5e4b476fd5356d3227
Jul 24 21:21:06.389885 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.389861    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:06.413333 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.413304    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:06.516725 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.516699    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"]
Jul 24 21:21:06.519357 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:21:06.519329    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25abda8_754d_48a7_85b3_6151343db03d.slice/crio-80a2e474e83b4910fc501da32c91f236e2400e7bb8c6e6552bdc2cb2ed444c82 WatchSource:0}: Error finding container 80a2e474e83b4910fc501da32c91f236e2400e7bb8c6e6552bdc2cb2ed444c82: Status 404 returned error can't find the container with id 80a2e474e83b4910fc501da32c91f236e2400e7bb8c6e6552bdc2cb2ed444c82
Jul 24 21:21:06.614404 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.614375    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:06.701061 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.701034    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss" event={"ID":"f25abda8-754d-48a7-85b3-6151343db03d","Type":"ContainerStarted","Data":"80a2e474e83b4910fc501da32c91f236e2400e7bb8c6e6552bdc2cb2ed444c82"}
Jul 24 21:21:06.702308 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.702281    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-4bb8x" event={"ID":"876f6180-7cf5-4176-813b-af69e63f3f2e","Type":"ContainerStarted","Data":"a3e48d217dd854d0aac6a11dfd2c58a891ce2fbde7e95e5e4b476fd5356d3227"}
Jul 24 21:21:06.713904 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.713867    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:06.813543 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:06.813475    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:10.721449 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:10.721409    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss" event={"ID":"f25abda8-754d-48a7-85b3-6151343db03d","Type":"ContainerStarted","Data":"522468454b9bac7c2e94e70d971b4898379a2033f2f5cafa869eb9079f55eca2"}
Jul 24 21:21:10.721910 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:10.721534    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:10.722879 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:10.722854    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-4bb8x" event={"ID":"876f6180-7cf5-4176-813b-af69e63f3f2e","Type":"ContainerStarted","Data":"a1d1efad6ee8e79acb3fb2a4301c9647717cc6e2811ccfabdea4e785c3cd0ebe"}
Jul 24 21:21:10.722991 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:10.722902    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:21:10.739642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:10.739606    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss" podStartSLOduration=2.22397803 podStartE2EDuration="5.739593696s" totalImagesPullingTime="3.515615666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:21:10.739446724 +0000 UTC m=+351.932624662" watchObservedRunningTime="2026-07-24 21:21:10.739593696 +0000 UTC m=+351.932771634"
Jul 24 21:21:10.757531 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:10.757492    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-4bb8x" podStartSLOduration=2.035923475 podStartE2EDuration="5.757480382s" totalImagesPullingTime="3.721556907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:21:10.756736371 +0000 UTC m=+351.949914328" watchObservedRunningTime="2026-07-24 21:21:10.757480382 +0000 UTC m=+351.950658318"
Jul 24 21:21:21.731300 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:21.731268    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tq4ss"
Jul 24 21:21:31.729172 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:21:31.729145    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-4bb8x"
Jul 24 21:22:01.680616 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:01.680577    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:21.156659 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:21.156623    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:34.216000 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.215964    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:34.308172 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.308144    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-664687df58-l8t8j"]
Jul 24 21:22:34.310250 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.310228    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.314834 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.314816    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 24 21:22:34.315390 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.315370    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 24 21:22:34.315483 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.315429    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 24 21:22:34.316435 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.316416    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-6q98w\""
Jul 24 21:22:34.329409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.329387    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-l8t8j"]
Jul 24 21:22:34.353409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.353382    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:34.402891 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.402871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-kube-api-access-bwrth\") pod \"kserve-controller-manager-664687df58-l8t8j\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") " pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.403003 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.402901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-cert\") pod \"kserve-controller-manager-664687df58-l8t8j\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") " pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.482529 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.482458    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:34.503720 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.503696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-kube-api-access-bwrth\") pod \"kserve-controller-manager-664687df58-l8t8j\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") " pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.503815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.503733    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-cert\") pod \"kserve-controller-manager-664687df58-l8t8j\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") " pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.506237 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.506217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-cert\") pod \"kserve-controller-manager-664687df58-l8t8j\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") " pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.531526 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.531494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-kube-api-access-bwrth\") pod \"kserve-controller-manager-664687df58-l8t8j\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") " pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.594384 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.594362    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:34.620699 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.620681    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:34.746458 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:34.746437    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-l8t8j"]
Jul 24 21:22:34.748766 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:22:34.748724    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dedd427_b4a7_47e4_8fd9_b6f80563f3da.slice/crio-efaf68a56d5292dc01dcf0a99992e009fdf95dd1d367efc5fb3c059b96c50798 WatchSource:0}: Error finding container efaf68a56d5292dc01dcf0a99992e009fdf95dd1d367efc5fb3c059b96c50798: Status 404 returned error can't find the container with id efaf68a56d5292dc01dcf0a99992e009fdf95dd1d367efc5fb3c059b96c50798
Jul 24 21:22:35.006006 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:35.005917    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-l8t8j" event={"ID":"0dedd427-b4a7-47e4-8fd9-b6f80563f3da","Type":"ContainerStarted","Data":"efaf68a56d5292dc01dcf0a99992e009fdf95dd1d367efc5fb3c059b96c50798"}
Jul 24 21:22:35.080024 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:35.079996    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:35.181085 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:35.181054    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:38.019804 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:38.019760    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-l8t8j" event={"ID":"0dedd427-b4a7-47e4-8fd9-b6f80563f3da","Type":"ContainerStarted","Data":"5ecf1ae96cc6cd15930fa87bf2bf59db59b9ab3c2f84587b4a59bd9df1e85835"}
Jul 24 21:22:38.020237 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:38.019853    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:22:38.039524 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:38.039474    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-664687df58-l8t8j" podStartSLOduration=1.648979517 podStartE2EDuration="4.03946201s" totalImagesPullingTime="2.390482493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:22:38.038658421 +0000 UTC m=+439.231836358" watchObservedRunningTime="2026-07-24 21:22:38.03946201 +0000 UTC m=+439.232639946"
Jul 24 21:22:55.285788 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:22:55.285700    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:09.028417 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:09.028386    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:23:17.940703 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:17.940649    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-664687df58-l8t8j"]
Jul 24 21:23:17.941229 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:17.940895    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-664687df58-l8t8j" podUID="0dedd427-b4a7-47e4-8fd9-b6f80563f3da" containerName="manager" containerID="cri-o://5ecf1ae96cc6cd15930fa87bf2bf59db59b9ab3c2f84587b4a59bd9df1e85835" gracePeriod=10
Jul 24 21:23:17.964865 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:17.964834    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-664687df58-pr42s"]
Jul 24 21:23:17.968046 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:17.968031    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:17.978586 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:17.978563    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-pr42s"]
Jul 24 21:23:18.111121 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.111093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9gp\" (UniqueName: \"kubernetes.io/projected/bddba108-8a4b-4745-94cd-7808b5aa308b-kube-api-access-fd9gp\") pod \"kserve-controller-manager-664687df58-pr42s\" (UID: \"bddba108-8a4b-4745-94cd-7808b5aa308b\") " pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.111239 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.111212    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bddba108-8a4b-4745-94cd-7808b5aa308b-cert\") pod \"kserve-controller-manager-664687df58-pr42s\" (UID: \"bddba108-8a4b-4745-94cd-7808b5aa308b\") " pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.162067 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.162040    2585 generic.go:356] "Generic (PLEG): container finished" podID="0dedd427-b4a7-47e4-8fd9-b6f80563f3da" containerID="5ecf1ae96cc6cd15930fa87bf2bf59db59b9ab3c2f84587b4a59bd9df1e85835" exitCode=0
Jul 24 21:23:18.162203 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.162113    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-l8t8j" event={"ID":"0dedd427-b4a7-47e4-8fd9-b6f80563f3da","Type":"ContainerDied","Data":"5ecf1ae96cc6cd15930fa87bf2bf59db59b9ab3c2f84587b4a59bd9df1e85835"}
Jul 24 21:23:18.182378 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.182353    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:23:18.212228 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.212161    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9gp\" (UniqueName: \"kubernetes.io/projected/bddba108-8a4b-4745-94cd-7808b5aa308b-kube-api-access-fd9gp\") pod \"kserve-controller-manager-664687df58-pr42s\" (UID: \"bddba108-8a4b-4745-94cd-7808b5aa308b\") " pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.212338 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.212227    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bddba108-8a4b-4745-94cd-7808b5aa308b-cert\") pod \"kserve-controller-manager-664687df58-pr42s\" (UID: \"bddba108-8a4b-4745-94cd-7808b5aa308b\") " pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.214730 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.214705    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bddba108-8a4b-4745-94cd-7808b5aa308b-cert\") pod \"kserve-controller-manager-664687df58-pr42s\" (UID: \"bddba108-8a4b-4745-94cd-7808b5aa308b\") " pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.220572 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.220552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9gp\" (UniqueName: \"kubernetes.io/projected/bddba108-8a4b-4745-94cd-7808b5aa308b-kube-api-access-fd9gp\") pod \"kserve-controller-manager-664687df58-pr42s\" (UID: \"bddba108-8a4b-4745-94cd-7808b5aa308b\") " pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.313035 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.312994    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-kube-api-access-bwrth\") pod \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") "
Jul 24 21:23:18.313182 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.313057    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-cert\") pod \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\" (UID: \"0dedd427-b4a7-47e4-8fd9-b6f80563f3da\") "
Jul 24 21:23:18.315315 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.315290    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-cert" (OuterVolumeSpecName: "cert") pod "0dedd427-b4a7-47e4-8fd9-b6f80563f3da" (UID: "0dedd427-b4a7-47e4-8fd9-b6f80563f3da"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:23:18.315416 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.315327    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-kube-api-access-bwrth" (OuterVolumeSpecName: "kube-api-access-bwrth") pod "0dedd427-b4a7-47e4-8fd9-b6f80563f3da" (UID: "0dedd427-b4a7-47e4-8fd9-b6f80563f3da"). InnerVolumeSpecName "kube-api-access-bwrth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:23:18.316412 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.316399    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:18.414297 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.414269    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrth\" (UniqueName: \"kubernetes.io/projected/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-kube-api-access-bwrth\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:23:18.414297 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.414296    2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dedd427-b4a7-47e4-8fd9-b6f80563f3da-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:23:18.437966 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:18.437906    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-pr42s"]
Jul 24 21:23:18.440010 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:23:18.439987    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddba108_8a4b_4745_94cd_7808b5aa308b.slice/crio-31255119a621ebf09d62534dc99829db8b98dfa95840bace7b70784638160c62 WatchSource:0}: Error finding container 31255119a621ebf09d62534dc99829db8b98dfa95840bace7b70784638160c62: Status 404 returned error can't find the container with id 31255119a621ebf09d62534dc99829db8b98dfa95840bace7b70784638160c62
Jul 24 21:23:19.166573 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.166545    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-l8t8j"
Jul 24 21:23:19.167014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.166539    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-l8t8j" event={"ID":"0dedd427-b4a7-47e4-8fd9-b6f80563f3da","Type":"ContainerDied","Data":"efaf68a56d5292dc01dcf0a99992e009fdf95dd1d367efc5fb3c059b96c50798"}
Jul 24 21:23:19.167014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.166702    2585 scope.go:117] "RemoveContainer" containerID="5ecf1ae96cc6cd15930fa87bf2bf59db59b9ab3c2f84587b4a59bd9df1e85835"
Jul 24 21:23:19.168203 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.168177    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-pr42s" event={"ID":"bddba108-8a4b-4745-94cd-7808b5aa308b","Type":"ContainerStarted","Data":"10def967562108401f434263147e722f64cc25a63a51e2820c05d8a3c6d7fca7"}
Jul 24 21:23:19.168328 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.168227    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-pr42s" event={"ID":"bddba108-8a4b-4745-94cd-7808b5aa308b","Type":"ContainerStarted","Data":"31255119a621ebf09d62534dc99829db8b98dfa95840bace7b70784638160c62"}
Jul 24 21:23:19.168328 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.168277    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:19.204308 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.204234    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-664687df58-pr42s" podStartSLOduration=1.716925672 podStartE2EDuration="2.204216949s" totalImagesPullingTime="487.291277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:23:19.204138045 +0000 UTC m=+480.397315982" watchObservedRunningTime="2026-07-24 21:23:19.204216949 +0000 UTC m=+480.397394887"
Jul 24 21:23:19.243021 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.242994    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-664687df58-l8t8j"]
Jul 24 21:23:19.244073 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.244055    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-664687df58-l8t8j"]
Jul 24 21:23:19.326479 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:19.326443    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dedd427-b4a7-47e4-8fd9-b6f80563f3da" path="/var/lib/kubelet/pods/0dedd427-b4a7-47e4-8fd9-b6f80563f3da/volumes"
Jul 24 21:23:21.954961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:21.954920    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:23.010443 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:23.010403    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:23.110719 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:23.110686    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:23.182465 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:23.182440    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:23.755274 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:23.755234    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:23.851959 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:23.851932    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.631222 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:33.631188    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.757758 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:33.757727    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.857037 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:33.857014    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:50.176012 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:50.175979    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-664687df58-pr42s"
Jul 24 21:23:51.262038 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.262006    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:51.317968 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.317935    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-76kg8"]
Jul 24 21:23:51.318254 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.318238    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dedd427-b4a7-47e4-8fd9-b6f80563f3da" containerName="manager"
Jul 24 21:23:51.318331 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.318257    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dedd427-b4a7-47e4-8fd9-b6f80563f3da" containerName="manager"
Jul 24 21:23:51.318331 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.318325    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dedd427-b4a7-47e4-8fd9-b6f80563f3da" containerName="manager"
Jul 24 21:23:51.321120 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.321101    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.323974 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.323951    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 24 21:23:51.323974 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.323957    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-m2pn5\""
Jul 24 21:23:51.336295 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.336270    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-76kg8"]
Jul 24 21:23:51.344761 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.344729    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8qs\" (UniqueName: \"kubernetes.io/projected/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-kube-api-access-bk8qs\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.344850 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.344773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-cert\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.360139 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.360103    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:51.440519 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.440498    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:51.445584 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.445558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8qs\" (UniqueName: \"kubernetes.io/projected/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-kube-api-access-bk8qs\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.445700 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.445591    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-cert\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.445758 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:23:51.445715    2585 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 21:23:51.445799 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:23:51.445783    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-cert podName:aebee1e6-2c9b-4cb6-9a69-ad6d209f5153 nodeName:}" failed. No retries permitted until 2026-07-24 21:23:51.945764683 +0000 UTC m=+513.138942598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-cert") pod "odh-model-controller-75d9cdbddf-76kg8" (UID: "aebee1e6-2c9b-4cb6-9a69-ad6d209f5153") : secret "odh-model-controller-webhook-cert" not found
Jul 24 21:23:51.457626 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.457598    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8qs\" (UniqueName: \"kubernetes.io/projected/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-kube-api-access-bk8qs\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.541470 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.541418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:51.949217 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.949190    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-cert\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:51.951897 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:51.951860    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aebee1e6-2c9b-4cb6-9a69-ad6d209f5153-cert\") pod \"odh-model-controller-75d9cdbddf-76kg8\" (UID: \"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153\") " pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:52.164074 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:52.164039    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:52.231943 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:52.231872    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:52.264819 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:52.264775    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:52.360858 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:52.360835    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-76kg8"]
Jul 24 21:23:52.362429 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:23:52.362406    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebee1e6_2c9b_4cb6_9a69_ad6d209f5153.slice/crio-fd0d0deb8460c19899632a0568adb49dec9193423d34aea569313b0bf7462c48 WatchSource:0}: Error finding container fd0d0deb8460c19899632a0568adb49dec9193423d34aea569313b0bf7462c48: Status 404 returned error can't find the container with id fd0d0deb8460c19899632a0568adb49dec9193423d34aea569313b0bf7462c48
Jul 24 21:23:53.288784 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:53.288747    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-76kg8" event={"ID":"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153","Type":"ContainerStarted","Data":"fd0d0deb8460c19899632a0568adb49dec9193423d34aea569313b0bf7462c48"}
Jul 24 21:23:55.298345 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:55.298309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-76kg8" event={"ID":"aebee1e6-2c9b-4cb6-9a69-ad6d209f5153","Type":"ContainerStarted","Data":"ecc2e00a0cdc9b5048e2a5eef74f8cd1e7beb92936307563871cbd469f2cc6cd"}
Jul 24 21:23:55.298786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:55.298443    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:23:55.322849 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:23:55.322805    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-76kg8" podStartSLOduration=2.104896101 podStartE2EDuration="4.322790869s" totalImagesPullingTime="2.217894768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:23:55.320698932 +0000 UTC m=+516.513876894" watchObservedRunningTime="2026-07-24 21:23:55.322790869 +0000 UTC m=+516.515968806"
Jul 24 21:24:06.305547 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:24:06.305515    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-76kg8"
Jul 24 21:24:09.893167 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:24:09.893076    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:26.258919 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:24:26.258881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:26.765547 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.765515    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:26.770701 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.770677    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"]
Jul 24 21:25:26.772518 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.772505    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:26.775872 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.775853    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 24 21:25:26.775961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.775932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d9dxm\""
Jul 24 21:25:26.783259 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.783231    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"]
Jul 24 21:25:26.914320 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.914286    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4v9\" (UniqueName: \"kubernetes.io/projected/d5dc306f-57b1-426f-a5fd-64adf8313deb-kube-api-access-6r4v9\") pod \"seaweedfs-tls-custom-6644bf5d6c-w78gv\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:26.914461 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:26.914333    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc306f-57b1-426f-a5fd-64adf8313deb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-w78gv\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:27.014828 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.014796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4v9\" (UniqueName: \"kubernetes.io/projected/d5dc306f-57b1-426f-a5fd-64adf8313deb-kube-api-access-6r4v9\") pod \"seaweedfs-tls-custom-6644bf5d6c-w78gv\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:27.014942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.014833    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc306f-57b1-426f-a5fd-64adf8313deb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-w78gv\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:27.015183 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.015166    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc306f-57b1-426f-a5fd-64adf8313deb-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-w78gv\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:27.024203 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.024139    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4v9\" (UniqueName: \"kubernetes.io/projected/d5dc306f-57b1-426f-a5fd-64adf8313deb-kube-api-access-6r4v9\") pod \"seaweedfs-tls-custom-6644bf5d6c-w78gv\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:27.081648 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.081626    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:27.204190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.204164    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"]
Jul 24 21:25:27.206653 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:25:27.206630    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dc306f_57b1_426f_a5fd_64adf8313deb.slice/crio-89602ad0f748b9b695bb9e9fe50de8fc6fd5cedd40f682804574ba1b0545234e WatchSource:0}: Error finding container 89602ad0f748b9b695bb9e9fe50de8fc6fd5cedd40f682804574ba1b0545234e: Status 404 returned error can't find the container with id 89602ad0f748b9b695bb9e9fe50de8fc6fd5cedd40f682804574ba1b0545234e
Jul 24 21:25:27.615801 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:27.615768    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv" event={"ID":"d5dc306f-57b1-426f-a5fd-64adf8313deb","Type":"ContainerStarted","Data":"89602ad0f748b9b695bb9e9fe50de8fc6fd5cedd40f682804574ba1b0545234e"}
Jul 24 21:25:30.627315 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:30.627283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv" event={"ID":"d5dc306f-57b1-426f-a5fd-64adf8313deb","Type":"ContainerStarted","Data":"cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f"}
Jul 24 21:25:30.644620 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:30.644563    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv" podStartSLOduration=1.793158759 podStartE2EDuration="4.64454597s" totalImagesPullingTime="2.851387211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:25:30.643514332 +0000 UTC m=+611.836692270" watchObservedRunningTime="2026-07-24 21:25:30.64454597 +0000 UTC m=+611.837723907"
Jul 24 21:25:31.979575 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:31.979544    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"]
Jul 24 21:25:32.634704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:32.634639    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv" podUID="d5dc306f-57b1-426f-a5fd-64adf8313deb" containerName="seaweedfs-tls-custom" containerID="cri-o://cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f" gracePeriod=30
Jul 24 21:25:33.860705 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.860684    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:33.863768 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.863748    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r4v9\" (UniqueName: \"kubernetes.io/projected/d5dc306f-57b1-426f-a5fd-64adf8313deb-kube-api-access-6r4v9\") pod \"d5dc306f-57b1-426f-a5fd-64adf8313deb\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") "
Jul 24 21:25:33.863889 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.863816    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc306f-57b1-426f-a5fd-64adf8313deb-data\") pod \"d5dc306f-57b1-426f-a5fd-64adf8313deb\" (UID: \"d5dc306f-57b1-426f-a5fd-64adf8313deb\") "
Jul 24 21:25:33.865310 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.865285    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dc306f-57b1-426f-a5fd-64adf8313deb-data" (OuterVolumeSpecName: "data") pod "d5dc306f-57b1-426f-a5fd-64adf8313deb" (UID: "d5dc306f-57b1-426f-a5fd-64adf8313deb"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:25:33.865958 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.865935    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dc306f-57b1-426f-a5fd-64adf8313deb-kube-api-access-6r4v9" (OuterVolumeSpecName: "kube-api-access-6r4v9") pod "d5dc306f-57b1-426f-a5fd-64adf8313deb" (UID: "d5dc306f-57b1-426f-a5fd-64adf8313deb"). InnerVolumeSpecName "kube-api-access-6r4v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:25:33.964823 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.964759    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4v9\" (UniqueName: \"kubernetes.io/projected/d5dc306f-57b1-426f-a5fd-64adf8313deb-kube-api-access-6r4v9\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:25:33.964823 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:33.964781    2585 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5dc306f-57b1-426f-a5fd-64adf8313deb-data\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:25:34.645052 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.645014    2585 generic.go:356] "Generic (PLEG): container finished" podID="d5dc306f-57b1-426f-a5fd-64adf8313deb" containerID="cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f" exitCode=0
Jul 24 21:25:34.645257 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.645073    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"
Jul 24 21:25:34.645257 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.645102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv" event={"ID":"d5dc306f-57b1-426f-a5fd-64adf8313deb","Type":"ContainerDied","Data":"cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f"}
Jul 24 21:25:34.645257 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.645163    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv" event={"ID":"d5dc306f-57b1-426f-a5fd-64adf8313deb","Type":"ContainerDied","Data":"89602ad0f748b9b695bb9e9fe50de8fc6fd5cedd40f682804574ba1b0545234e"}
Jul 24 21:25:34.645257 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.645185    2585 scope.go:117] "RemoveContainer" containerID="cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f"
Jul 24 21:25:34.655973 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.655953    2585 scope.go:117] "RemoveContainer" containerID="cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f"
Jul 24 21:25:34.656248 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:25:34.656227    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f\": container with ID starting with cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f not found: ID does not exist" containerID="cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f"
Jul 24 21:25:34.656292 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.656257    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f"} err="failed to get container status \"cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f\": rpc error: code = NotFound desc = could not find container \"cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f\": container with ID starting with cdb436c0f410daf1058ff1176480a51afe3cc177da6c12f5598704764418806f not found: ID does not exist"
Jul 24 21:25:34.666633 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.666610    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"]
Jul 24 21:25:34.670464 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:34.670442    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-w78gv"]
Jul 24 21:25:35.327353 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:35.327324    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5dc306f-57b1-426f-a5fd-64adf8313deb" path="/var/lib/kubelet/pods/d5dc306f-57b1-426f-a5fd-64adf8313deb/volumes"
Jul 24 21:25:48.024820 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:48.024708    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:49.211972 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:25:49.211933    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:05.547749 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:26:05.547714    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:05.645409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:26:05.645032    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:17.790411 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:26:17.790375    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:18.136348 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:26:18.136318    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:19.904895 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:27:19.904812    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:28:03.089735 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:28:03.089694    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:28:03.163818 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:28:03.163780    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:28:03.659267 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:28:03.659231    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.302688 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:29:38.302626    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.388344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:29:38.388309    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.897657 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:29:38.897621    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:58.299844 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:29:58.299796    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:58.518012 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:29:58.517972    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:30:20.882797 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:30:20.882758    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:23.614134 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:31:23.614096    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:24.780125 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:31:24.780085    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:39.905790 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:31:39.905709    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:34:20.421418 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:34:20.421383    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:34:25.524821 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:34:25.524777    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:34:25.744086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:34:25.744054    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:55.851634 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:35:55.851576    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:31.981963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:36:31.981934    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:37.126692 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:36:37.126634    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:37.748083 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:36:37.748048    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:36:54.011712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:36:54.011658    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:34.348497 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:38:34.348459    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:39.267612 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:38:39.267562    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:39.645282 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:38:39.645237    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:38:44.685319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:38:44.685270    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:56.492012 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:40:56.491983    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:41:00.941459 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:41:00.941424    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:41:01.085496 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:41:01.085433    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:41:43.534388 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:41:43.534355    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:41:48.645030 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:41:48.644999    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:41:48.976583 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:41:48.976478    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:42:23.927946 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:42:23.927909    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:42:29.101899 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:42:29.101867    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:42:29.214448 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:42:29.214402    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:42:29.310263 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:42:29.310228    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:42:29.404532 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:42:29.404501    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:59.315590 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:43:59.315514    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:59.403494 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:43:59.403464    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:44:04.325045 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:44:04.325008    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:44:04.401181 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:44:04.401144    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:44:04.468233 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:44:04.468203    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:44:04.526944 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:44:04.526913    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:45:34.496600 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.496556    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:45:34.600936 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.600902    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"]
Jul 24 21:45:34.601217 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.601204    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5dc306f-57b1-426f-a5fd-64adf8313deb" containerName="seaweedfs-tls-custom"
Jul 24 21:45:34.601260 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.601219    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5dc306f-57b1-426f-a5fd-64adf8313deb" containerName="seaweedfs-tls-custom"
Jul 24 21:45:34.601294 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.601272    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5dc306f-57b1-426f-a5fd-64adf8313deb" containerName="seaweedfs-tls-custom"
Jul 24 21:45:34.604219 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.604204    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.606962 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.606938    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 24 21:45:34.607169 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.606983    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:45:34.607169 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.606992    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:45:34.607169 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.607017    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:45:34.607169 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.607159    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 24 21:45:34.608301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.608283    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:45:34.608409 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.608291    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:45:34.616154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.616132    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"]
Jul 24 21:45:34.733188 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.733162    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.733312 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.733193    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8qc\" (UniqueName: \"kubernetes.io/projected/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-api-access-jx8qc\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.733312 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.733213    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87587ea5-335a-4ca9-b795-7dcbe3b68f13-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.733312 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.733237    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.733312 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.733294    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.833852 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.833781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.833852 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.833819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8qc\" (UniqueName: \"kubernetes.io/projected/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-api-access-jx8qc\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.833852 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.833844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87587ea5-335a-4ca9-b795-7dcbe3b68f13-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.834084 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.833896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.834084 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.833947    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.834337 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.834318    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.834550 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.834494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.834550 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.834506    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.836389 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.836369    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87587ea5-335a-4ca9-b795-7dcbe3b68f13-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.842711 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.842689    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8qc\" (UniqueName: \"kubernetes.io/projected/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-api-access-jx8qc\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:34.914657 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:34.914635    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:35.044623 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:35.044531    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"]
Jul 24 21:45:35.046813 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:45:35.046785    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87587ea5_335a_4ca9_b795_7dcbe3b68f13.slice/crio-a90edafc247715d671a25c3dc5db4117f185d1296c26ce1fb6c856f885235b67 WatchSource:0}: Error finding container a90edafc247715d671a25c3dc5db4117f185d1296c26ce1fb6c856f885235b67: Status 404 returned error can't find the container with id a90edafc247715d671a25c3dc5db4117f185d1296c26ce1fb6c856f885235b67
Jul 24 21:45:35.048996 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:35.048978    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:45:35.866292 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:35.866256    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerStarted","Data":"a90edafc247715d671a25c3dc5db4117f185d1296c26ce1fb6c856f885235b67"}
Jul 24 21:45:39.887754 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:39.887714    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerStarted","Data":"c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3"}
Jul 24 21:45:42.899076 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:42.899043    2585 generic.go:356] "Generic (PLEG): container finished" podID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerID="c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3" exitCode=0
Jul 24 21:45:42.899425 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:42.899119    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerDied","Data":"c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3"}
Jul 24 21:45:55.962110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:55.962022    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerStarted","Data":"9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda"}
Jul 24 21:45:58.978114 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:58.978077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerStarted","Data":"6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b"}
Jul 24 21:45:58.978500 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:58.978210    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:59.000616 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:59.000557    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podStartSLOduration=2.347098511 podStartE2EDuration="25.000544455s" totalImagesPullingTime="18.800147944s" totalInitContainerRuntime="3.853298s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:45:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:45:58.999091612 +0000 UTC m=+1840.192269569" watchObservedRunningTime="2026-07-24 21:45:59.000544455 +0000 UTC m=+1840.193722395"
Jul 24 21:45:59.983252 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:59.983223    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:45:59.984446 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:45:59.984405    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:00.986610 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:00.986559    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:05.990917 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:05.990889    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:46:05.991416 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:05.991386    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:15.991991 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:15.991952    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:25.991387 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:25.991342    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:35.991400 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:35.991359    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:45.992187 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:45.992146    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:46:55.991443 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:46:55.991401    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:47:05.991659 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:05.991609    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:47:15.325891 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:15.325864    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:47:24.737738 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:24.737692    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"]
Jul 24 21:47:24.738202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:24.737973    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" containerID="cri-o://9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda" gracePeriod=30
Jul 24 21:47:24.738202 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:24.738080    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kube-rbac-proxy" containerID="cri-o://6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b" gracePeriod=30
Jul 24 21:47:24.821222 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:24.821184    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:25.291805 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:25.291773    2585 generic.go:356] "Generic (PLEG): container finished" podID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerID="6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b" exitCode=2
Jul 24 21:47:25.291972 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:25.291825    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerDied","Data":"6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b"}
Jul 24 21:47:25.322221 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:25.322188    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 24 21:47:25.987364 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:25.987312    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 24 21:47:28.983169 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:28.983143    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:47:29.160239 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160192    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-rbac-proxy-sar-config\") pod \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") "
Jul 24 21:47:29.160432 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160299    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kserve-provision-location\") pod \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") "
Jul 24 21:47:29.160432 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160358    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-cabundle-cert\") pod \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") "
Jul 24 21:47:29.160432 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160383    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87587ea5-335a-4ca9-b795-7dcbe3b68f13-proxy-tls\") pod \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") "
Jul 24 21:47:29.160595 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160448    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx8qc\" (UniqueName: \"kubernetes.io/projected/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-api-access-jx8qc\") pod \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\" (UID: \"87587ea5-335a-4ca9-b795-7dcbe3b68f13\") "
Jul 24 21:47:29.160711 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160654    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87587ea5-335a-4ca9-b795-7dcbe3b68f13" (UID: "87587ea5-335a-4ca9-b795-7dcbe3b68f13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:47:29.160759 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160705    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87587ea5-335a-4ca9-b795-7dcbe3b68f13" (UID: "87587ea5-335a-4ca9-b795-7dcbe3b68f13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:29.160793 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.160748    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87587ea5-335a-4ca9-b795-7dcbe3b68f13" (UID: "87587ea5-335a-4ca9-b795-7dcbe3b68f13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:29.162831 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.162810    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87587ea5-335a-4ca9-b795-7dcbe3b68f13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87587ea5-335a-4ca9-b795-7dcbe3b68f13" (UID: "87587ea5-335a-4ca9-b795-7dcbe3b68f13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:47:29.162831 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.162819    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-api-access-jx8qc" (OuterVolumeSpecName: "kube-api-access-jx8qc") pod "87587ea5-335a-4ca9-b795-7dcbe3b68f13" (UID: "87587ea5-335a-4ca9-b795-7dcbe3b68f13"). InnerVolumeSpecName "kube-api-access-jx8qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:47:29.261642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.261607    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8qc\" (UniqueName: \"kubernetes.io/projected/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-api-access-jx8qc\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.261642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.261640    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.261642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.261649    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87587ea5-335a-4ca9-b795-7dcbe3b68f13-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.261886 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.261661    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87587ea5-335a-4ca9-b795-7dcbe3b68f13-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.261886 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.261694    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87587ea5-335a-4ca9-b795-7dcbe3b68f13-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:47:29.308161 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.308126    2585 generic.go:356] "Generic (PLEG): container finished" podID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerID="9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda" exitCode=0
Jul 24 21:47:29.308288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.308172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerDied","Data":"9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda"}
Jul 24 21:47:29.308288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.308191    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"
Jul 24 21:47:29.308288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.308223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd" event={"ID":"87587ea5-335a-4ca9-b795-7dcbe3b68f13","Type":"ContainerDied","Data":"a90edafc247715d671a25c3dc5db4117f185d1296c26ce1fb6c856f885235b67"}
Jul 24 21:47:29.308288 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.308249    2585 scope.go:117] "RemoveContainer" containerID="6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b"
Jul 24 21:47:29.317192 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.317177    2585 scope.go:117] "RemoveContainer" containerID="9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda"
Jul 24 21:47:29.324077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.324062    2585 scope.go:117] "RemoveContainer" containerID="c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3"
Jul 24 21:47:29.331989 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.331971    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"]
Jul 24 21:47:29.332127 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.332108    2585 scope.go:117] "RemoveContainer" containerID="6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b"
Jul 24 21:47:29.332357 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:47:29.332341    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b\": container with ID starting with 6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b not found: ID does not exist" containerID="6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b"
Jul 24 21:47:29.332407 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.332364    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b"} err="failed to get container status \"6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b\": rpc error: code = NotFound desc = could not find container \"6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b\": container with ID starting with 6223c3b6564c3d5d0374d1550e29ea18c6cab294a6fdd38c49e10ff54157237b not found: ID does not exist"
Jul 24 21:47:29.332407 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.332378    2585 scope.go:117] "RemoveContainer" containerID="9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda"
Jul 24 21:47:29.332616 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:47:29.332599    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda\": container with ID starting with 9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda not found: ID does not exist" containerID="9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda"
Jul 24 21:47:29.332656 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.332623    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda"} err="failed to get container status \"9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda\": rpc error: code = NotFound desc = could not find container \"9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda\": container with ID starting with 9c559a3ef36873d45bb98d83c44824fc814234b5ce0b57969222843aefde6bda not found: ID does not exist"
Jul 24 21:47:29.332656 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.332641    2585 scope.go:117] "RemoveContainer" containerID="c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3"
Jul 24 21:47:29.332978 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:47:29.332951    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3\": container with ID starting with c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3 not found: ID does not exist" containerID="c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3"
Jul 24 21:47:29.333016 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.332991    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3"} err="failed to get container status \"c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3\": rpc error: code = NotFound desc = could not find container \"c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3\": container with ID starting with c5520600d2c18985aecedc08f63e8571577ee9675cb5d069c6ebfc00c05694f3 not found: ID does not exist"
Jul 24 21:47:29.336379 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.336359    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-4jwtd"]
Jul 24 21:47:29.813396 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.813359    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:29.867590 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867560    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"]
Jul 24 21:47:29.867842 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867830    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container"
Jul 24 21:47:29.867881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867844    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container"
Jul 24 21:47:29.867881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867856    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="storage-initializer"
Jul 24 21:47:29.867881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867862    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="storage-initializer"
Jul 24 21:47:29.867881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867874    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kube-rbac-proxy"
Jul 24 21:47:29.867881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867880    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kube-rbac-proxy"
Jul 24 21:47:29.868036 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867935    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kube-rbac-proxy"
Jul 24 21:47:29.868036 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.867942    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" containerName="kserve-container"
Jul 24 21:47:29.872538 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.872501    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:29.872960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.872928    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:29.875378 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875351    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:47:29.875490 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875379    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:47:29.875560 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:47:29.875560 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:47:29.875560 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875549    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 24 21:47:29.875720 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875587    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 24 21:47:29.875888 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.875873    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:47:29.883123 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.883099    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"]
Jul 24 21:47:29.886795 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.886777    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"]
Jul 24 21:47:29.886923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.886905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:29.890268 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.890249    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 24 21:47:29.905261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:29.905243    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"]
Jul 24 21:47:30.051656 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.051623    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:30.067577 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067514    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.067712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067571    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.067712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067602    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.067712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566451e2-c746-4d74-9d73-a179fd0756d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.067712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067659    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.067712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067707    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.067894 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067738    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvrz\" (UniqueName: \"kubernetes.io/projected/566451e2-c746-4d74-9d73-a179fd0756d4-kube-api-access-brvrz\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.067894 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067768    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgvp\" (UniqueName: \"kubernetes.io/projected/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-api-access-bhgvp\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.067894 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067796    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/566451e2-c746-4d74-9d73-a179fd0756d4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.067894 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.067854    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.169147 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169109    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.169147 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvrz\" (UniqueName: \"kubernetes.io/projected/566451e2-c746-4d74-9d73-a179fd0756d4-kube-api-access-brvrz\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.169344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169183    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgvp\" (UniqueName: \"kubernetes.io/projected/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-api-access-bhgvp\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.169344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169209    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/566451e2-c746-4d74-9d73-a179fd0756d4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.169344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169240    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.169344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169279    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.169344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169328    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.169578 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169350    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.169578 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169376    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566451e2-c746-4d74-9d73-a179fd0756d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.169578 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169404    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.169750 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169694    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/566451e2-c746-4d74-9d73-a179fd0756d4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.169840 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.169811    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.170177 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.170152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.170435 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.170152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.170435 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.170266    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.170547 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.170494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.172355 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.172335    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.172490 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.172473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566451e2-c746-4d74-9d73-a179fd0756d4-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.179717 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.179697    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvrz\" (UniqueName: \"kubernetes.io/projected/566451e2-c746-4d74-9d73-a179fd0756d4-kube-api-access-brvrz\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-ll64b\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.179950 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.179927    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgvp\" (UniqueName: \"kubernetes.io/projected/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-api-access-bhgvp\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.185654 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.185621    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:30.196936 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.196917    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:30.214693 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.214639    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:30.328782 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.328759    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"]
Jul 24 21:47:30.330967 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:47:30.330936    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566451e2_c746_4d74_9d73_a179fd0756d4.slice/crio-5990ff84cb755b91a4a7ef7570b4040ec0b692a9d38019931a54fbd3d6c5a016 WatchSource:0}: Error finding container 5990ff84cb755b91a4a7ef7570b4040ec0b692a9d38019931a54fbd3d6c5a016: Status 404 returned error can't find the container with id 5990ff84cb755b91a4a7ef7570b4040ec0b692a9d38019931a54fbd3d6c5a016
Jul 24 21:47:30.351012 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:30.350991    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"]
Jul 24 21:47:30.357726 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:47:30.357705    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872fe0e6_ba06_4cd8_bf2d_19d5e666e249.slice/crio-ddcdda3ff83db81462335ddbb608e207db9f676929b55e1121ae3fc0fff07bb5 WatchSource:0}: Error finding container ddcdda3ff83db81462335ddbb608e207db9f676929b55e1121ae3fc0fff07bb5: Status 404 returned error can't find the container with id ddcdda3ff83db81462335ddbb608e207db9f676929b55e1121ae3fc0fff07bb5
Jul 24 21:47:31.319297 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:31.319256    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerStarted","Data":"df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217"}
Jul 24 21:47:31.319717 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:31.319358    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerStarted","Data":"ddcdda3ff83db81462335ddbb608e207db9f676929b55e1121ae3fc0fff07bb5"}
Jul 24 21:47:31.320813 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:31.320788    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerStarted","Data":"25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb"}
Jul 24 21:47:31.320890 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:31.320847    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerStarted","Data":"5990ff84cb755b91a4a7ef7570b4040ec0b692a9d38019931a54fbd3d6c5a016"}
Jul 24 21:47:31.326814 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:31.326790    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87587ea5-335a-4ca9-b795-7dcbe3b68f13" path="/var/lib/kubelet/pods/87587ea5-335a-4ca9-b795-7dcbe3b68f13/volumes"
Jul 24 21:47:34.332343 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:34.332309    2585 generic.go:356] "Generic (PLEG): container finished" podID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerID="df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217" exitCode=0
Jul 24 21:47:34.332741 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:34.332382    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerDied","Data":"df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217"}
Jul 24 21:47:34.334046 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:34.334008    2585 generic.go:356] "Generic (PLEG): container finished" podID="566451e2-c746-4d74-9d73-a179fd0756d4" containerID="25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb" exitCode=0
Jul 24 21:47:34.334224 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:34.334093    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerDied","Data":"25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb"}
Jul 24 21:47:35.339624 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.339581    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerStarted","Data":"247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486"}
Jul 24 21:47:35.340059 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.339656    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerStarted","Data":"2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963"}
Jul 24 21:47:35.340059 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.339880    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:35.341644 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.341618    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerStarted","Data":"eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21"}
Jul 24 21:47:35.341770 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.341688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerStarted","Data":"08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210"}
Jul 24 21:47:35.342009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.341990    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:35.342085 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.342044    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:35.343300 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.343276    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:47:35.359890 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.359849    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podStartSLOduration=2.579733196 podStartE2EDuration="6.359838357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780105161s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:47:35.358255148 +0000 UTC m=+1936.551433115" watchObservedRunningTime="2026-07-24 21:47:35.359838357 +0000 UTC m=+1936.553016331"
Jul 24 21:47:35.376410 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:35.376372    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podStartSLOduration=2.651603235 podStartE2EDuration="6.376359974s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.724756739s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:47:35.375235967 +0000 UTC m=+1936.568413945" watchObservedRunningTime="2026-07-24 21:47:35.376359974 +0000 UTC m=+1936.569537910"
Jul 24 21:47:36.345387 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:36.345346    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:47:36.345828 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:36.345660    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:36.346814 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:36.346790    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:47:37.349440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:37.349406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:47:41.350439 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:41.350400    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:47:41.351022 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:41.350986    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:47:42.353994 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:42.353963    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:47:42.354412 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:42.354384    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:47:51.350976 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:51.350936    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:47:52.354394 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:47:52.354355    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:48:01.351906 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:01.351861    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:48:02.354759 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:02.354721    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:48:11.350930 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:11.350838    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:48:12.354924 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:12.354879    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:48:21.351639 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:21.351597    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:48:22.354390 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:22.354343    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:48:31.351283 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:31.351241    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:48:32.355094 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:32.355048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:48:41.351685 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:41.351623    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:48:42.355218 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:42.355176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:48:51.352534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:51.352505    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:48:52.355864 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:52.355836    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:48:59.916960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:59.916927    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"]
Jul 24 21:48:59.917461 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:59.917177    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kube-rbac-proxy" containerID="cri-o://247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486" gracePeriod=30
Jul 24 21:48:59.917461 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:59.917166    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" containerID="cri-o://2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963" gracePeriod=30
Jul 24 21:48:59.935249 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:48:59.935222    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:49:00.662971 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:00.662937    2585 generic.go:356] "Generic (PLEG): container finished" podID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerID="247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486" exitCode=2
Jul 24 21:49:00.663197 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:00.662994    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerDied","Data":"247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486"}
Jul 24 21:49:02.350609 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:02.350566    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 24 21:49:02.354931 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:02.354900    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 24 21:49:04.251528 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.251506    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:49:04.268945 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.268922    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-cabundle-cert\") pod \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") "
Jul 24 21:49:04.269023 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.268959    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-proxy-tls\") pod \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") "
Jul 24 21:49:04.269023 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.268979    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kserve-provision-location\") pod \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") "
Jul 24 21:49:04.269146 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.269035    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhgvp\" (UniqueName: \"kubernetes.io/projected/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-api-access-bhgvp\") pod \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") "
Jul 24 21:49:04.269146 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.269087    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-rbac-proxy-sar-config\") pod \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\" (UID: \"872fe0e6-ba06-4cd8-bf2d-19d5e666e249\") "
Jul 24 21:49:04.269315 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.269282    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "872fe0e6-ba06-4cd8-bf2d-19d5e666e249" (UID: "872fe0e6-ba06-4cd8-bf2d-19d5e666e249"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:49:04.269315 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.269307    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "872fe0e6-ba06-4cd8-bf2d-19d5e666e249" (UID: "872fe0e6-ba06-4cd8-bf2d-19d5e666e249"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:49:04.269527 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.269499    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "872fe0e6-ba06-4cd8-bf2d-19d5e666e249" (UID: "872fe0e6-ba06-4cd8-bf2d-19d5e666e249"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:49:04.271161 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.271140    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "872fe0e6-ba06-4cd8-bf2d-19d5e666e249" (UID: "872fe0e6-ba06-4cd8-bf2d-19d5e666e249"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:49:04.271243 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.271167    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-api-access-bhgvp" (OuterVolumeSpecName: "kube-api-access-bhgvp") pod "872fe0e6-ba06-4cd8-bf2d-19d5e666e249" (UID: "872fe0e6-ba06-4cd8-bf2d-19d5e666e249"). InnerVolumeSpecName "kube-api-access-bhgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:49:04.370165 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.370145    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:49:04.370165 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.370170    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:49:04.370298 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.370180    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:49:04.370298 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.370189    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgvp\" (UniqueName: \"kubernetes.io/projected/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-api-access-bhgvp\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:49:04.370298 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.370198    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872fe0e6-ba06-4cd8-bf2d-19d5e666e249-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:49:04.682440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.682347    2585 generic.go:356] "Generic (PLEG): container finished" podID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerID="2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963" exitCode=0
Jul 24 21:49:04.682440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.682384    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerDied","Data":"2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963"}
Jul 24 21:49:04.682440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.682405    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"
Jul 24 21:49:04.682440 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.682434    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2" event={"ID":"872fe0e6-ba06-4cd8-bf2d-19d5e666e249","Type":"ContainerDied","Data":"ddcdda3ff83db81462335ddbb608e207db9f676929b55e1121ae3fc0fff07bb5"}
Jul 24 21:49:04.682787 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.682459    2585 scope.go:117] "RemoveContainer" containerID="247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486"
Jul 24 21:49:04.690759 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.690743    2585 scope.go:117] "RemoveContainer" containerID="2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963"
Jul 24 21:49:04.698081 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.698065    2585 scope.go:117] "RemoveContainer" containerID="df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217"
Jul 24 21:49:04.704090 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.704070    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"]
Jul 24 21:49:04.705628 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.705604    2585 scope.go:117] "RemoveContainer" containerID="247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486"
Jul 24 21:49:04.706059 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:49:04.706014    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486\": container with ID starting with 247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486 not found: ID does not exist" containerID="247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486"
Jul 24 21:49:04.706163 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.706062    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486"} err="failed to get container status \"247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486\": rpc error: code = NotFound desc = could not find container \"247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486\": container with ID starting with 247ab8b37d1ef92e06b4ceead71f2649323598bd5f90e6cf54f3791615e2c486 not found: ID does not exist"
Jul 24 21:49:04.706163 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.706087    2585 scope.go:117] "RemoveContainer" containerID="2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963"
Jul 24 21:49:04.706393 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:49:04.706375    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963\": container with ID starting with 2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963 not found: ID does not exist" containerID="2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963"
Jul 24 21:49:04.706480 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.706403    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963"} err="failed to get container status \"2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963\": rpc error: code = NotFound desc = could not find container \"2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963\": container with ID starting with 2142b2dd5fdd7b06e8952ee1d31c9f788370eacd11226faf3a58a80cdecff963 not found: ID does not exist"
Jul 24 21:49:04.706480 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.706425    2585 scope.go:117] "RemoveContainer" containerID="df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217"
Jul 24 21:49:04.706743 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:49:04.706720    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217\": container with ID starting with df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217 not found: ID does not exist" containerID="df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217"
Jul 24 21:49:04.706845 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.706748    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217"} err="failed to get container status \"df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217\": rpc error: code = NotFound desc = could not find container \"df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217\": container with ID starting with df7db15c112685709ac9d9fb857b641ed2bf9e7ccc611d15b5b355fea7aaf217 not found: ID does not exist"
Jul 24 21:49:04.707918 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:04.707898    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-x6zp2"]
Jul 24 21:49:05.327261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:49:05.327226    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" path="/var/lib/kubelet/pods/872fe0e6-ba06-4cd8-bf2d-19d5e666e249/volumes"
Jul 24 21:50:30.174697 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:30.174649    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"]
Jul 24 21:50:30.175179 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:30.174929    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" containerID="cri-o://08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210" gracePeriod=30
Jul 24 21:50:30.175179 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:30.174970    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kube-rbac-proxy" containerID="cri-o://eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21" gracePeriod=30
Jul 24 21:50:30.225574 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:30.225528    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:30.995762 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:30.995734    2585 generic.go:356] "Generic (PLEG): container finished" podID="566451e2-c746-4d74-9d73-a179fd0756d4" containerID="eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21" exitCode=2
Jul 24 21:50:30.995928 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:30.995808    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerDied","Data":"eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21"}
Jul 24 21:50:31.345978 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:31.345895    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 24 21:50:31.351237 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:31.351212    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 24 21:50:34.422277 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.422255    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:50:34.501166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501103    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/566451e2-c746-4d74-9d73-a179fd0756d4-kserve-provision-location\") pod \"566451e2-c746-4d74-9d73-a179fd0756d4\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") "
Jul 24 21:50:34.501166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501163    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brvrz\" (UniqueName: \"kubernetes.io/projected/566451e2-c746-4d74-9d73-a179fd0756d4-kube-api-access-brvrz\") pod \"566451e2-c746-4d74-9d73-a179fd0756d4\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") "
Jul 24 21:50:34.501367 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501202    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566451e2-c746-4d74-9d73-a179fd0756d4-proxy-tls\") pod \"566451e2-c746-4d74-9d73-a179fd0756d4\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") "
Jul 24 21:50:34.501367 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501253    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-cabundle-cert\") pod \"566451e2-c746-4d74-9d73-a179fd0756d4\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") "
Jul 24 21:50:34.501367 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501290    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-kube-rbac-proxy-sar-config\") pod \"566451e2-c746-4d74-9d73-a179fd0756d4\" (UID: \"566451e2-c746-4d74-9d73-a179fd0756d4\") "
Jul 24 21:50:34.501528 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501479    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566451e2-c746-4d74-9d73-a179fd0756d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "566451e2-c746-4d74-9d73-a179fd0756d4" (UID: "566451e2-c746-4d74-9d73-a179fd0756d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:50:34.501705 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501653    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "566451e2-c746-4d74-9d73-a179fd0756d4" (UID: "566451e2-c746-4d74-9d73-a179fd0756d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:50:34.501815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.501734    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "566451e2-c746-4d74-9d73-a179fd0756d4" (UID: "566451e2-c746-4d74-9d73-a179fd0756d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:50:34.503441 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.503411    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566451e2-c746-4d74-9d73-a179fd0756d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "566451e2-c746-4d74-9d73-a179fd0756d4" (UID: "566451e2-c746-4d74-9d73-a179fd0756d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:50:34.503441 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.503418    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566451e2-c746-4d74-9d73-a179fd0756d4-kube-api-access-brvrz" (OuterVolumeSpecName: "kube-api-access-brvrz") pod "566451e2-c746-4d74-9d73-a179fd0756d4" (UID: "566451e2-c746-4d74-9d73-a179fd0756d4"). InnerVolumeSpecName "kube-api-access-brvrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:50:34.601912 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.601887    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:50:34.601912 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.601910    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/566451e2-c746-4d74-9d73-a179fd0756d4-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:50:34.602043 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.601919    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvrz\" (UniqueName: \"kubernetes.io/projected/566451e2-c746-4d74-9d73-a179fd0756d4-kube-api-access-brvrz\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:50:34.602043 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.601929    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/566451e2-c746-4d74-9d73-a179fd0756d4-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:50:34.602043 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:34.601938    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/566451e2-c746-4d74-9d73-a179fd0756d4-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:50:35.011785 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.011749    2585 generic.go:356] "Generic (PLEG): container finished" podID="566451e2-c746-4d74-9d73-a179fd0756d4" containerID="08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210" exitCode=0
Jul 24 21:50:35.011965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.011832    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"
Jul 24 21:50:35.011965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.011839    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerDied","Data":"08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210"}
Jul 24 21:50:35.011965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.011912    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b" event={"ID":"566451e2-c746-4d74-9d73-a179fd0756d4","Type":"ContainerDied","Data":"5990ff84cb755b91a4a7ef7570b4040ec0b692a9d38019931a54fbd3d6c5a016"}
Jul 24 21:50:35.011965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.011942    2585 scope.go:117] "RemoveContainer" containerID="eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21"
Jul 24 21:50:35.020858 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.020841    2585 scope.go:117] "RemoveContainer" containerID="08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210"
Jul 24 21:50:35.028362 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.028341    2585 scope.go:117] "RemoveContainer" containerID="25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb"
Jul 24 21:50:35.034496 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.034470    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"]
Jul 24 21:50:35.038122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.038100    2585 scope.go:117] "RemoveContainer" containerID="eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21"
Jul 24 21:50:35.038798 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:50:35.038619    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21\": container with ID starting with eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21 not found: ID does not exist" containerID="eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21"
Jul 24 21:50:35.038798 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.038658    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21"} err="failed to get container status \"eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21\": rpc error: code = NotFound desc = could not find container \"eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21\": container with ID starting with eb372a4dcca5f8b35f2e2adeac87053c4b1d3e9dd61dc1a377acdd7893596d21 not found: ID does not exist"
Jul 24 21:50:35.038798 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.038708    2585 scope.go:117] "RemoveContainer" containerID="08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210"
Jul 24 21:50:35.039537 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:50:35.039516    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210\": container with ID starting with 08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210 not found: ID does not exist" containerID="08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210"
Jul 24 21:50:35.039615 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.039543    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210"} err="failed to get container status \"08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210\": rpc error: code = NotFound desc = could not find container \"08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210\": container with ID starting with 08e810ef99c28c0fc9cadc46b498ec659ed996b67e27b378dcd73b8bba950210 not found: ID does not exist"
Jul 24 21:50:35.039615 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.039559    2585 scope.go:117] "RemoveContainer" containerID="25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb"
Jul 24 21:50:35.039878 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:50:35.039857    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb\": container with ID starting with 25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb not found: ID does not exist" containerID="25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb"
Jul 24 21:50:35.039930 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.039885    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb"} err="failed to get container status \"25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb\": rpc error: code = NotFound desc = could not find container \"25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb\": container with ID starting with 25a6a0666bea103f8fac7a4f77ec53f5c9a96608ee12199bb20292b4219daaeb not found: ID does not exist"
Jul 24 21:50:35.040503 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.040485    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-ll64b"]
Jul 24 21:50:35.210904 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.210861    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:35.271161 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271075    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"]
Jul 24 21:50:35.271384 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271369    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kube-rbac-proxy"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271389    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kube-rbac-proxy"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271411    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="storage-initializer"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271417    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="storage-initializer"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271423    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271428    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271435    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="storage-initializer"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271440    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="storage-initializer"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271449    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container"
Jul 24 21:50:35.271451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271453    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container"
Jul 24 21:50:35.271908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271466    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kube-rbac-proxy"
Jul 24 21:50:35.271908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271489    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kube-rbac-proxy"
Jul 24 21:50:35.271908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271531    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kube-rbac-proxy"
Jul 24 21:50:35.271908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271539    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kube-rbac-proxy"
Jul 24 21:50:35.271908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271551    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="872fe0e6-ba06-4cd8-bf2d-19d5e666e249" containerName="kserve-container"
Jul 24 21:50:35.271908 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.271560    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" containerName="kserve-container"
Jul 24 21:50:35.273718 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.273655    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:35.276172 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.276153    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"]
Jul 24 21:50:35.276294 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.276274    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.278933 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.278914    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 24 21:50:35.279189 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279166    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 24 21:50:35.279189 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279183    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:50:35.279349 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279235    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:50:35.279349 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279274    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:50:35.279349 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279183    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:50:35.279527 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279400    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:50:35.279896 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.279881    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.282191 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.282170    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 24 21:50:35.286046 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.286027    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"]
Jul 24 21:50:35.289552 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.289532    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"]
Jul 24 21:50:35.328354 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.328317    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566451e2-c746-4d74-9d73-a179fd0756d4" path="/var/lib/kubelet/pods/566451e2-c746-4d74-9d73-a179fd0756d4/volumes"
Jul 24 21:50:35.408593 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408555    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdgz\" (UniqueName: \"kubernetes.io/projected/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-api-access-8sdgz\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.408593 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408597    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.408848 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408619    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kserve-provision-location\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.408848 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.408848 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7lwx\" (UniqueName: \"kubernetes.io/projected/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-api-access-v7lwx\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.408848 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-cabundle-cert\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.408848 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408835    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd267891-fa83-49d9-af28-acd5491f5ecc-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.409110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.409110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd267891-fa83-49d9-af28-acd5491f5ecc-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.409110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.408903    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.489685 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.489611    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:35.509733 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509688    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdgz\" (UniqueName: \"kubernetes.io/projected/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-api-access-8sdgz\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.509881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509738    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.509881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509772    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kserve-provision-location\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.509881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.509881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509857    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7lwx\" (UniqueName: \"kubernetes.io/projected/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-api-access-v7lwx\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.510086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-cabundle-cert\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.510086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.509984    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd267891-fa83-49d9-af28-acd5491f5ecc-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.510086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.510086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510042    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd267891-fa83-49d9-af28-acd5491f5ecc-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.510086 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.510317 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kserve-provision-location\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.510700 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510409    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd267891-fa83-49d9-af28-acd5491f5ecc-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.510700 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:50:35.510539    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 24 21:50:35.510700 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510581    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.510700 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:50:35.510599    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls podName:66608fdc-3ff5-49eb-a6d5-5f54abc87594 nodeName:}" failed. No retries permitted until 2026-07-24 21:50:36.010581043 +0000 UTC m=+2117.203758977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls") pod "isvc-canary-stop-predictor-8bd7d86fd-jwvtx" (UID: "66608fdc-3ff5-49eb-a6d5-5f54abc87594") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 24 21:50:35.510700 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510618    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.511036 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510706    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-cabundle-cert\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.511036 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.510718    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.513140 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.513119    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd267891-fa83-49d9-af28-acd5491f5ecc-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.523731 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.523655    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdgz\" (UniqueName: \"kubernetes.io/projected/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-api-access-8sdgz\") pod \"isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.523835 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.523769    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7lwx\" (UniqueName: \"kubernetes.io/projected/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-api-access-v7lwx\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:35.598488 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.598452    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:35.632399 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.632362    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:35.730933 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:35.730910    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"]
Jul 24 21:50:35.732989 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:50:35.732955    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd267891_fa83_49d9_af28_acd5491f5ecc.slice/crio-24a55c5b434b591e25039a4299770cd4018eb7d14d90a863c7aaf752238f6cfb WatchSource:0}: Error finding container 24a55c5b434b591e25039a4299770cd4018eb7d14d90a863c7aaf752238f6cfb: Status 404 returned error can't find the container with id 24a55c5b434b591e25039a4299770cd4018eb7d14d90a863c7aaf752238f6cfb
Jul 24 21:50:36.013405 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:36.013373    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:36.016228 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:36.016189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-jwvtx\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:36.018327 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:36.018305    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerStarted","Data":"920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f"}
Jul 24 21:50:36.018405 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:36.018360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerStarted","Data":"24a55c5b434b591e25039a4299770cd4018eb7d14d90a863c7aaf752238f6cfb"}
Jul 24 21:50:36.190554 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:36.190525    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:36.318782 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:36.318750    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"]
Jul 24 21:50:36.322764 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:50:36.322732    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66608fdc_3ff5_49eb_a6d5_5f54abc87594.slice/crio-29f1b0493ec61459c023edc263b22e2cdd134e0db1b649c9fc12eb0c48ffa027 WatchSource:0}: Error finding container 29f1b0493ec61459c023edc263b22e2cdd134e0db1b649c9fc12eb0c48ffa027: Status 404 returned error can't find the container with id 29f1b0493ec61459c023edc263b22e2cdd134e0db1b649c9fc12eb0c48ffa027
Jul 24 21:50:37.023803 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:37.023764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerStarted","Data":"2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e"}
Jul 24 21:50:37.024160 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:37.023832    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerStarted","Data":"29f1b0493ec61459c023edc263b22e2cdd134e0db1b649c9fc12eb0c48ffa027"}
Jul 24 21:50:37.880252 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:37.880219    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:40.038217 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:40.038182    2585 generic.go:356] "Generic (PLEG): container finished" podID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerID="920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f" exitCode=0
Jul 24 21:50:40.038573 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:40.038222    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerDied","Data":"920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f"}
Jul 24 21:50:41.043070 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.043037    2585 generic.go:356] "Generic (PLEG): container finished" podID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerID="2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e" exitCode=0
Jul 24 21:50:41.043492 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.043112    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerDied","Data":"2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e"}
Jul 24 21:50:41.045215 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.045194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerStarted","Data":"d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2"}
Jul 24 21:50:41.045311 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.045237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerStarted","Data":"45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f"}
Jul 24 21:50:41.047886 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.047868    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:41.047960 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.047922    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:41.049410 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.049386    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:50:41.100175 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:41.100125    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podStartSLOduration=2.374769077 podStartE2EDuration="6.100106381s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.725337304s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:50:41.099732942 +0000 UTC m=+2122.292910908" watchObservedRunningTime="2026-07-24 21:50:41.100106381 +0000 UTC m=+2122.293284320"
Jul 24 21:50:42.050611 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:42.050575    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerStarted","Data":"4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d"}
Jul 24 21:50:42.051091 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:42.050632    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerStarted","Data":"edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb"}
Jul 24 21:50:42.051091 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:42.050831    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:50:42.051091 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:42.051050    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:42.073317 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:42.073275    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podStartSLOduration=3.368506622 podStartE2EDuration="7.073260235s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.704753613s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:50:42.071090285 +0000 UTC m=+2123.264268248" watchObservedRunningTime="2026-07-24 21:50:42.073260235 +0000 UTC m=+2123.266438228"
Jul 24 21:50:43.055035 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:43.054998    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:43.056324 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:43.056295    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:50:44.059148 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:44.059109    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:50:47.055985 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:47.055952    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:50:47.056441 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:47.056408    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:50:49.063915 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:49.063887    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:50:49.064422 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:49.064394    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:50:57.056928 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:57.056883    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:50:59.065209 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:50:59.065165    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:51:07.056915 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:07.056872    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:51:09.065294 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:09.065253    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:51:17.056946 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:17.056862    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:51:19.064372 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:19.064331    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:51:27.056370 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:27.056330    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:51:29.064361 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:29.064322    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:51:37.056639 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:37.056600    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:51:39.064680 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:39.064622    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:51:47.056976 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:47.056934    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:51:49.064500 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:49.064461    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:51:57.057832 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:57.057802    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:51:59.065335 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:51:59.065305    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:52:01.803050 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:01.803002    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:52:05.306447 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.306416    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"]
Jul 24 21:52:05.306931 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.306739    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" containerID="cri-o://edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb" gracePeriod=30
Jul 24 21:52:05.306931 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.306812    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kube-rbac-proxy" containerID="cri-o://4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d" gracePeriod=30
Jul 24 21:52:05.317235 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.317204    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:52:05.342772 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.342732    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"]
Jul 24 21:52:05.343014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.342944    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" containerID="cri-o://45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f" gracePeriod=30
Jul 24 21:52:05.343014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.343005    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kube-rbac-proxy" containerID="cri-o://d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2" gracePeriod=30
Jul 24 21:52:05.375319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:05.375290    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:52:06.370141 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:06.370110    2585 generic.go:356] "Generic (PLEG): container finished" podID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerID="4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d" exitCode=2
Jul 24 21:52:06.370526 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:06.370183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerDied","Data":"4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d"}
Jul 24 21:52:06.372267 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:06.372249    2585 generic.go:356] "Generic (PLEG): container finished" podID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerID="d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2" exitCode=2
Jul 24 21:52:06.372341 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:06.372279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerDied","Data":"d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2"}
Jul 24 21:52:07.051944 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:07.051898    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 24 21:52:07.057323 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:07.057298    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 24 21:52:09.059679 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.059623    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 24 21:52:09.065045 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.065021    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 24 21:52:09.642491 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.642470    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:52:09.702275 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702246    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-rbac-proxy-sar-config\") pod \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") "
Jul 24 21:52:09.702387 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702283    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-cabundle-cert\") pod \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") "
Jul 24 21:52:09.702387 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702313    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7lwx\" (UniqueName: \"kubernetes.io/projected/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-api-access-v7lwx\") pod \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") "
Jul 24 21:52:09.702387 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kserve-provision-location\") pod \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") "
Jul 24 21:52:09.702573 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702389    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls\") pod \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\" (UID: \"66608fdc-3ff5-49eb-a6d5-5f54abc87594\") "
Jul 24 21:52:09.702727 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702659    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66608fdc-3ff5-49eb-a6d5-5f54abc87594" (UID: "66608fdc-3ff5-49eb-a6d5-5f54abc87594"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:52:09.702727 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702703    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66608fdc-3ff5-49eb-a6d5-5f54abc87594" (UID: "66608fdc-3ff5-49eb-a6d5-5f54abc87594"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:52:09.702919 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.702761    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66608fdc-3ff5-49eb-a6d5-5f54abc87594" (UID: "66608fdc-3ff5-49eb-a6d5-5f54abc87594"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:52:09.704630 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.704606    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-api-access-v7lwx" (OuterVolumeSpecName: "kube-api-access-v7lwx") pod "66608fdc-3ff5-49eb-a6d5-5f54abc87594" (UID: "66608fdc-3ff5-49eb-a6d5-5f54abc87594"). InnerVolumeSpecName "kube-api-access-v7lwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:52:09.704630 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.704615    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66608fdc-3ff5-49eb-a6d5-5f54abc87594" (UID: "66608fdc-3ff5-49eb-a6d5-5f54abc87594"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:52:09.803190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.803160    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66608fdc-3ff5-49eb-a6d5-5f54abc87594-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:09.803190 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.803188    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:09.803344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.803199    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66608fdc-3ff5-49eb-a6d5-5f54abc87594-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:09.803344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.803209    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7lwx\" (UniqueName: \"kubernetes.io/projected/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kube-api-access-v7lwx\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:09.803344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.803217    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66608fdc-3ff5-49eb-a6d5-5f54abc87594-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:09.866464 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.866434    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:52:09.904049 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904025    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sdgz\" (UniqueName: \"kubernetes.io/projected/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-api-access-8sdgz\") pod \"bd267891-fa83-49d9-af28-acd5491f5ecc\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") "
Jul 24 21:52:09.904229 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904056    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd267891-fa83-49d9-af28-acd5491f5ecc-kserve-provision-location\") pod \"bd267891-fa83-49d9-af28-acd5491f5ecc\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") "
Jul 24 21:52:09.904229 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904074    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-cabundle-cert\") pod \"bd267891-fa83-49d9-af28-acd5491f5ecc\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") "
Jul 24 21:52:09.904229 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904156    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-rbac-proxy-sar-config\") pod \"bd267891-fa83-49d9-af28-acd5491f5ecc\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") "
Jul 24 21:52:09.904229 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904182    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd267891-fa83-49d9-af28-acd5491f5ecc-proxy-tls\") pod \"bd267891-fa83-49d9-af28-acd5491f5ecc\" (UID: \"bd267891-fa83-49d9-af28-acd5491f5ecc\") "
Jul 24 21:52:09.904451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904370    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd267891-fa83-49d9-af28-acd5491f5ecc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd267891-fa83-49d9-af28-acd5491f5ecc" (UID: "bd267891-fa83-49d9-af28-acd5491f5ecc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:52:09.904550 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904525    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd267891-fa83-49d9-af28-acd5491f5ecc" (UID: "bd267891-fa83-49d9-af28-acd5491f5ecc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:52:09.904550 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.904532    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd267891-fa83-49d9-af28-acd5491f5ecc" (UID: "bd267891-fa83-49d9-af28-acd5491f5ecc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:52:09.906192 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.906172    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-api-access-8sdgz" (OuterVolumeSpecName: "kube-api-access-8sdgz") pod "bd267891-fa83-49d9-af28-acd5491f5ecc" (UID: "bd267891-fa83-49d9-af28-acd5491f5ecc"). InnerVolumeSpecName "kube-api-access-8sdgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:52:09.906310 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:09.906290    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd267891-fa83-49d9-af28-acd5491f5ecc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd267891-fa83-49d9-af28-acd5491f5ecc" (UID: "bd267891-fa83-49d9-af28-acd5491f5ecc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:52:10.004843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.004795    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sdgz\" (UniqueName: \"kubernetes.io/projected/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-api-access-8sdgz\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:10.004843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.004816    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd267891-fa83-49d9-af28-acd5491f5ecc-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:10.004843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.004825    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:10.004843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.004833    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd267891-fa83-49d9-af28-acd5491f5ecc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:10.004843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.004842    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd267891-fa83-49d9-af28-acd5491f5ecc-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:52:10.389811 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.389775    2585 generic.go:356] "Generic (PLEG): container finished" podID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerID="45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f" exitCode=0
Jul 24 21:52:10.390374 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.389835    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"
Jul 24 21:52:10.390374 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.389868    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerDied","Data":"45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f"}
Jul 24 21:52:10.390374 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.389938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd" event={"ID":"bd267891-fa83-49d9-af28-acd5491f5ecc","Type":"ContainerDied","Data":"24a55c5b434b591e25039a4299770cd4018eb7d14d90a863c7aaf752238f6cfb"}
Jul 24 21:52:10.390374 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.389977    2585 scope.go:117] "RemoveContainer" containerID="d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2"
Jul 24 21:52:10.392038 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.392019    2585 generic.go:356] "Generic (PLEG): container finished" podID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerID="edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb" exitCode=0
Jul 24 21:52:10.392134 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.392047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerDied","Data":"edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb"}
Jul 24 21:52:10.392134 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.392092    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx" event={"ID":"66608fdc-3ff5-49eb-a6d5-5f54abc87594","Type":"ContainerDied","Data":"29f1b0493ec61459c023edc263b22e2cdd134e0db1b649c9fc12eb0c48ffa027"}
Jul 24 21:52:10.392134 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.392105    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"
Jul 24 21:52:10.398353 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.398330    2585 scope.go:117] "RemoveContainer" containerID="45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f"
Jul 24 21:52:10.406545 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.406529    2585 scope.go:117] "RemoveContainer" containerID="920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f"
Jul 24 21:52:10.413965 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.413952    2585 scope.go:117] "RemoveContainer" containerID="d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2"
Jul 24 21:52:10.414226 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:10.414210    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2\": container with ID starting with d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2 not found: ID does not exist" containerID="d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2"
Jul 24 21:52:10.414278 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.414233    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2"} err="failed to get container status \"d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2\": rpc error: code = NotFound desc = could not find container \"d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2\": container with ID starting with d0cf1f375cd822ecbc0184e1d13949322fc5bdb353ba895d4d296e880306d8a2 not found: ID does not exist"
Jul 24 21:52:10.414278 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.414249    2585 scope.go:117] "RemoveContainer" containerID="45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f"
Jul 24 21:52:10.414504 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:10.414487    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f\": container with ID starting with 45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f not found: ID does not exist" containerID="45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f"
Jul 24 21:52:10.414544 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.414512    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f"} err="failed to get container status \"45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f\": rpc error: code = NotFound desc = could not find container \"45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f\": container with ID starting with 45a306fd178405111f811d5f49efc446d9fb94a0a9b64f9828eda93c0b729b0f not found: ID does not exist"
Jul 24 21:52:10.414544 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.414532    2585 scope.go:117] "RemoveContainer" containerID="920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f"
Jul 24 21:52:10.414861 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:10.414833    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f\": container with ID starting with 920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f not found: ID does not exist" containerID="920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f"
Jul 24 21:52:10.414923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.414865    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f"} err="failed to get container status \"920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f\": rpc error: code = NotFound desc = could not find container \"920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f\": container with ID starting with 920c937fafb7ee8eaaa2ff07126b48256553e5d79bf8194d9b738103a1cfcb4f not found: ID does not exist"
Jul 24 21:52:10.414923 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.414878    2585 scope.go:117] "RemoveContainer" containerID="4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d"
Jul 24 21:52:10.415466 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.415436    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"]
Jul 24 21:52:10.420534 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.420514    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b66dd5859-cwjnd"]
Jul 24 21:52:10.422250 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.422236    2585 scope.go:117] "RemoveContainer" containerID="edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb"
Jul 24 21:52:10.429251 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.429232    2585 scope.go:117] "RemoveContainer" containerID="2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e"
Jul 24 21:52:10.432079 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.432057    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"]
Jul 24 21:52:10.435849 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.435830    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-jwvtx"]
Jul 24 21:52:10.436925 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.436911    2585 scope.go:117] "RemoveContainer" containerID="4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d"
Jul 24 21:52:10.437181 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:10.437163    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d\": container with ID starting with 4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d not found: ID does not exist" containerID="4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d"
Jul 24 21:52:10.437244 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.437188    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d"} err="failed to get container status \"4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d\": rpc error: code = NotFound desc = could not find container \"4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d\": container with ID starting with 4188e31ece474e32e418558249250064309d7692b9ae15374d313e04a1d3704d not found: ID does not exist"
Jul 24 21:52:10.437244 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.437204    2585 scope.go:117] "RemoveContainer" containerID="edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb"
Jul 24 21:52:10.437412 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:10.437396    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb\": container with ID starting with edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb not found: ID does not exist" containerID="edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb"
Jul 24 21:52:10.437454 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.437416    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb"} err="failed to get container status \"edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb\": rpc error: code = NotFound desc = could not find container \"edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb\": container with ID starting with edd832c439d72c2fc3eb11b7d084d58566d79b76af25a59f4ee440de36c97feb not found: ID does not exist"
Jul 24 21:52:10.437454 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.437427    2585 scope.go:117] "RemoveContainer" containerID="2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e"
Jul 24 21:52:10.437643 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:10.437625    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e\": container with ID starting with 2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e not found: ID does not exist" containerID="2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e"
Jul 24 21:52:10.437709 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:10.437649    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e"} err="failed to get container status \"2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e\": rpc error: code = NotFound desc = could not find container \"2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e\": container with ID starting with 2318ebe1804b4090f2167fe0c2f9bcedefaa8b0fb9549ed347facbdedd209e3e not found: ID does not exist"
Jul 24 21:52:11.327196 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:11.327159    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" path="/var/lib/kubelet/pods/66608fdc-3ff5-49eb-a6d5-5f54abc87594/volumes"
Jul 24 21:52:11.327631 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:11.327618    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" path="/var/lib/kubelet/pods/bd267891-fa83-49d9-af28-acd5491f5ecc/volumes"
Jul 24 21:52:15.539149 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.539085    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:52:15.598009 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.597979    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"]
Jul 24 21:52:15.598308 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598291    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kube-rbac-proxy"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598313    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kube-rbac-proxy"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598323    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="storage-initializer"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598329    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="storage-initializer"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598339    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598345    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598351    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kube-rbac-proxy"
Jul 24 21:52:15.598357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598357    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kube-rbac-proxy"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598363    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598368    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598381    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="storage-initializer"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598390    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="storage-initializer"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598438    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kserve-container"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598446    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd267891-fa83-49d9-af28-acd5491f5ecc" containerName="kube-rbac-proxy"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598451    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kserve-container"
Jul 24 21:52:15.598562 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.598459    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="66608fdc-3ff5-49eb-a6d5-5f54abc87594" containerName="kube-rbac-proxy"
Jul 24 21:52:15.603239 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.603218    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.606751 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.606729    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:52:15.606751 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.606754    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 24 21:52:15.606949 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.606728    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:52:15.607758 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.607740    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:52:15.607856 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.607754    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:52:15.607856 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.607769    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 24 21:52:15.607974 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.607797    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:52:15.612767 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.612747    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"]
Jul 24 21:52:15.640188 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.640168    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.640299 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.640194    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.640299 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.640230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.640299 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.640282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.640402 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.640304    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrc8\" (UniqueName: \"kubernetes.io/projected/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-api-access-xjrc8\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.740844 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.740820    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrc8\" (UniqueName: \"kubernetes.io/projected/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-api-access-xjrc8\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.740957 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.740860    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.740957 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.740876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.740957 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.740915    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.740957 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.740948    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.741145 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:15.741090    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 24 21:52:15.741196 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:52:15.741163    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls podName:62ee8567-6871-4613-9d74-0dd3bfa3e85d nodeName:}" failed. No retries permitted until 2026-07-24 21:52:16.241141204 +0000 UTC m=+2217.434319137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-kndbs" (UID: "62ee8567-6871-4613-9d74-0dd3bfa3e85d") : secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 24 21:52:15.741307 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.741290    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.741598 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.741578    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.741688 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.741612    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.752355 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.752337    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrc8\" (UniqueName: \"kubernetes.io/projected/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-api-access-xjrc8\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:15.852122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:15.852052    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:52:16.245634 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:16.245601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:16.248130 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:16.248110    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-kndbs\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:16.515055 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:16.514970    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:16.646253 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:16.646227    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"]
Jul 24 21:52:16.648036 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:52:16.648013    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ee8567_6871_4613_9d74_0dd3bfa3e85d.slice/crio-732237d9fe4235490bab068dd4305c437099fda33798b73852f044fa28ff3eba WatchSource:0}: Error finding container 732237d9fe4235490bab068dd4305c437099fda33798b73852f044fa28ff3eba: Status 404 returned error can't find the container with id 732237d9fe4235490bab068dd4305c437099fda33798b73852f044fa28ff3eba
Jul 24 21:52:17.419286 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:17.419247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerStarted","Data":"bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45"}
Jul 24 21:52:17.419475 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:17.419302    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerStarted","Data":"732237d9fe4235490bab068dd4305c437099fda33798b73852f044fa28ff3eba"}
Jul 24 21:52:21.434225 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:21.434186    2585 generic.go:356] "Generic (PLEG): container finished" podID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerID="bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45" exitCode=0
Jul 24 21:52:21.434589 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:21.434260    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerDied","Data":"bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45"}
Jul 24 21:52:21.435582 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:21.435566    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:52:28.465124 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:28.465088    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerStarted","Data":"c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa"}
Jul 24 21:52:28.465526 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:28.465156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerStarted","Data":"1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c"}
Jul 24 21:52:28.465526 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:28.465351    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:28.488697 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:28.488621    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podStartSLOduration=2.939926401 podStartE2EDuration="13.488601856s" totalImagesPullingTime="6.497737716s" totalInitContainerRuntime="4.050937739s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:52:28.486943736 +0000 UTC m=+2229.680121674" watchObservedRunningTime="2026-07-24 21:52:28.488601856 +0000 UTC m=+2229.681779791"
Jul 24 21:52:29.469196 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:29.469164    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:29.470485 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:29.470460    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:52:30.472956 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:30.472911    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:52:35.478037 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:35.478000    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:52:35.478633 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:35.478605    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:52:45.479522 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:45.479440    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:52:55.478610 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:52:55.478571    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:53:05.478756 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:05.478712    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:53:15.479251 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:15.479206    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:53:25.479084 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:25.479047    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:53:35.478873 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:35.478832    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 24 21:53:39.706474 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:39.706442    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:53:45.479873 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:45.479838    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:53:55.703737 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:55.703702    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"]
Jul 24 21:53:55.704283 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:55.704014    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container" containerID="cri-o://1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c" gracePeriod=30
Jul 24 21:53:55.704283 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:55.704023    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kube-rbac-proxy" containerID="cri-o://c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa" gracePeriod=30
Jul 24 21:53:55.725811 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:55.725782    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:53:56.788207 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:56.788172    2585 generic.go:356] "Generic (PLEG): container finished" podID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerID="c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa" exitCode=2
Jul 24 21:53:56.788565 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:53:56.788241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerDied","Data":"c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa"}
Jul 24 21:54:00.547894 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.547866    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:54:00.597052 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597022    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls\") pod \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") "
Jul 24 21:54:00.597178 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597063    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kserve-provision-location\") pod \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") "
Jul 24 21:54:00.597178 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597104    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjrc8\" (UniqueName: \"kubernetes.io/projected/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-api-access-xjrc8\") pod \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") "
Jul 24 21:54:00.597178 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597123    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-rbac-proxy-sar-config\") pod \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") "
Jul 24 21:54:00.597178 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597158    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-cabundle-cert\") pod \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\" (UID: \"62ee8567-6871-4613-9d74-0dd3bfa3e85d\") "
Jul 24 21:54:00.597504 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597478    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62ee8567-6871-4613-9d74-0dd3bfa3e85d" (UID: "62ee8567-6871-4613-9d74-0dd3bfa3e85d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:54:00.597504 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597482    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62ee8567-6871-4613-9d74-0dd3bfa3e85d" (UID: "62ee8567-6871-4613-9d74-0dd3bfa3e85d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:54:00.597597 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.597530    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62ee8567-6871-4613-9d74-0dd3bfa3e85d" (UID: "62ee8567-6871-4613-9d74-0dd3bfa3e85d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:54:00.599265 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.599247    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-api-access-xjrc8" (OuterVolumeSpecName: "kube-api-access-xjrc8") pod "62ee8567-6871-4613-9d74-0dd3bfa3e85d" (UID: "62ee8567-6871-4613-9d74-0dd3bfa3e85d"). InnerVolumeSpecName "kube-api-access-xjrc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:54:00.599346 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.599331    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62ee8567-6871-4613-9d74-0dd3bfa3e85d" (UID: "62ee8567-6871-4613-9d74-0dd3bfa3e85d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:54:00.698166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.698107    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:54:00.698166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.698133    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62ee8567-6871-4613-9d74-0dd3bfa3e85d-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:54:00.698166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.698143    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:54:00.698166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.698153    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrc8\" (UniqueName: \"kubernetes.io/projected/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-api-access-xjrc8\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:54:00.698166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.698162    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62ee8567-6871-4613-9d74-0dd3bfa3e85d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:54:00.805319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.805285    2585 generic.go:356] "Generic (PLEG): container finished" podID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerID="1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c" exitCode=0
Jul 24 21:54:00.805555 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.805341    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerDied","Data":"1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c"}
Jul 24 21:54:00.805555 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.805408    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" event={"ID":"62ee8567-6871-4613-9d74-0dd3bfa3e85d","Type":"ContainerDied","Data":"732237d9fe4235490bab068dd4305c437099fda33798b73852f044fa28ff3eba"}
Jul 24 21:54:00.805555 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.805451    2585 scope.go:117] "RemoveContainer" containerID="c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa"
Jul 24 21:54:00.805555 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.805345    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"
Jul 24 21:54:00.813266 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.813218    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:54:00.815921 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.815896    2585 scope.go:117] "RemoveContainer" containerID="1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c"
Jul 24 21:54:00.824843 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.824825    2585 scope.go:117] "RemoveContainer" containerID="bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45"
Jul 24 21:54:00.830836 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.830815    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"]
Jul 24 21:54:00.832954 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.832939    2585 scope.go:117] "RemoveContainer" containerID="c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa"
Jul 24 21:54:00.833186 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:54:00.833156    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa\": container with ID starting with c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa not found: ID does not exist" containerID="c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa"
Jul 24 21:54:00.833242 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.833190    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa"} err="failed to get container status \"c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa\": rpc error: code = NotFound desc = could not find container \"c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa\": container with ID starting with c5028f6b4b768d10cde55de580231dccf5a9bdeaa9beb7b5fc384dc8995f3caa not found: ID does not exist"
Jul 24 21:54:00.833242 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.833211    2585 scope.go:117] "RemoveContainer" containerID="1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c"
Jul 24 21:54:00.833447 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:54:00.833433    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c\": container with ID starting with 1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c not found: ID does not exist" containerID="1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c"
Jul 24 21:54:00.833506 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.833452    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c"} err="failed to get container status \"1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c\": rpc error: code = NotFound desc = could not find container \"1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c\": container with ID starting with 1deb096ab034252cdd78336a23bfd56277d139e864779cfae5c0d8a98118973c not found: ID does not exist"
Jul 24 21:54:00.833506 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.833464    2585 scope.go:117] "RemoveContainer" containerID="bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45"
Jul 24 21:54:00.833708 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:54:00.833690    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45\": container with ID starting with bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45 not found: ID does not exist" containerID="bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45"
Jul 24 21:54:00.833766 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.833716    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45"} err="failed to get container status \"bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45\": rpc error: code = NotFound desc = could not find container \"bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45\": container with ID starting with bda4bdbf0f149e03a324e6b5e8a46f07526908ccb0d9f0b20bd8c6c034db0c45 not found: ID does not exist"
Jul 24 21:54:00.837626 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.837606    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs"]
Jul 24 21:54:00.881813 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.881782    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"]
Jul 24 21:54:00.882083 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882066    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container"
Jul 24 21:54:00.882158 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882087    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container"
Jul 24 21:54:00.882158 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882107    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="storage-initializer"
Jul 24 21:54:00.882158 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882116    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="storage-initializer"
Jul 24 21:54:00.882158 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882135    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kube-rbac-proxy"
Jul 24 21:54:00.882158 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882143    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kube-rbac-proxy"
Jul 24 21:54:00.882400 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882216    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kserve-container"
Jul 24 21:54:00.882400 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.882229    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kube-rbac-proxy"
Jul 24 21:54:00.886406 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.886388    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:00.890411 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890390    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:54:00.890515 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890499    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:54:00.890574 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890529    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:54:00.890632 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890587    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 24 21:54:00.890632 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890593    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 24 21:54:00.890757 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890624    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:54:00.890877 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.890864    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:54:00.898718 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.898698    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"]
Jul 24 21:54:00.903851 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.901988    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:00.903851 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.902067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:00.903851 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.902107    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnm2x\" (UniqueName: \"kubernetes.io/projected/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-api-access-pnm2x\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:00.903851 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.902141    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:00.903851 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.902204    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:00.952076 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:00.952009    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:54:01.003196 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.003165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnm2x\" (UniqueName: \"kubernetes.io/projected/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-api-access-pnm2x\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.003301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.003203    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.003354 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.003331    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.003408 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.003392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.003469 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.003454    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.003738 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.003714    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.004043 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.004022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.004179 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.004103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.005896 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.005879    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.011249 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.011230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnm2x\" (UniqueName: \"kubernetes.io/projected/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-api-access-pnm2x\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.195964 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.195936    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:01.325575 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:54:01.325549    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b31cb0_ad5b_42c8_b22a_480ba6b0efe1.slice/crio-314fa16fb7ffe48ee2e0877007c0f77d1b37de4273991496329fafdfa5f6c3dd WatchSource:0}: Error finding container 314fa16fb7ffe48ee2e0877007c0f77d1b37de4273991496329fafdfa5f6c3dd: Status 404 returned error can't find the container with id 314fa16fb7ffe48ee2e0877007c0f77d1b37de4273991496329fafdfa5f6c3dd
Jul 24 21:54:01.326791 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.326765    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" path="/var/lib/kubelet/pods/62ee8567-6871-4613-9d74-0dd3bfa3e85d/volumes"
Jul 24 21:54:01.327116 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.327103    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"]
Jul 24 21:54:01.473210 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.473128    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-kndbs" podUID="62ee8567-6871-4613-9d74-0dd3bfa3e85d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": context deadline exceeded"
Jul 24 21:54:01.810961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.810864    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerStarted","Data":"9f9d92840cc933fcaf2c978b4b503710dd0d849bbaf31b4baf209525b3b069c3"}
Jul 24 21:54:01.810961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:01.810930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerStarted","Data":"314fa16fb7ffe48ee2e0877007c0f77d1b37de4273991496329fafdfa5f6c3dd"}
Jul 24 21:54:05.826971 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:05.826935    2585 generic.go:356] "Generic (PLEG): container finished" podID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerID="9f9d92840cc933fcaf2c978b4b503710dd0d849bbaf31b4baf209525b3b069c3" exitCode=0
Jul 24 21:54:05.827364 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:05.826999    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerDied","Data":"9f9d92840cc933fcaf2c978b4b503710dd0d849bbaf31b4baf209525b3b069c3"}
Jul 24 21:54:06.832951 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:06.832905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerStarted","Data":"2b843b9610f3db1d5e53d32c7eee2668b294738172c0c24ce83054358e16d828"}
Jul 24 21:54:06.833376 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:06.832981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerStarted","Data":"2031595610b0d93e95437e65bdfce7344653ccd4c1b5a595a6946df402085760"}
Jul 24 21:54:06.833376 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:06.833274    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:06.833376 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:06.833337    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:06.834410 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:06.834388    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:54:06.853356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:06.853304    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podStartSLOduration=2.87860304 podStartE2EDuration="6.853290817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.974687777s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:54:06.851111366 +0000 UTC m=+2328.044289315" watchObservedRunningTime="2026-07-24 21:54:06.853290817 +0000 UTC m=+2328.046468754"
Jul 24 21:54:07.838385 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:07.838345    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:54:12.843264 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:12.843187    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:54:12.843637 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:12.843611    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:54:22.844231 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:22.844186    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:54:32.844654 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:32.844608    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:54:42.844542 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:42.844500    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:54:52.844285 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:54:52.844237    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:55:02.844447 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:02.844406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:55:12.844272 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:12.844233    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:55:22.845254 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:22.845225    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:55:28.503813 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:28.503774    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:31.614874 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:31.614835    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"]
Jul 24 21:55:31.615260 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:31.615059    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" containerID="cri-o://2031595610b0d93e95437e65bdfce7344653ccd4c1b5a595a6946df402085760" gracePeriod=30
Jul 24 21:55:31.615260 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:31.615106    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kube-rbac-proxy" containerID="cri-o://2b843b9610f3db1d5e53d32c7eee2668b294738172c0c24ce83054358e16d828" gracePeriod=30
Jul 24 21:55:31.626922 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:31.626895    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:32.136726 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:32.136692    2585 generic.go:356] "Generic (PLEG): container finished" podID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerID="2b843b9610f3db1d5e53d32c7eee2668b294738172c0c24ce83054358e16d828" exitCode=2
Jul 24 21:55:32.136901 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:32.136765    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerDied","Data":"2b843b9610f3db1d5e53d32c7eee2668b294738172c0c24ce83054358e16d828"}
Jul 24 21:55:32.839584 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:32.839545    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 24 21:55:32.843881 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:32.843857    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 24 21:55:36.152603 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.152559    2585 generic.go:356] "Generic (PLEG): container finished" podID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerID="2031595610b0d93e95437e65bdfce7344653ccd4c1b5a595a6946df402085760" exitCode=0
Jul 24 21:55:36.152964 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.152624    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerDied","Data":"2031595610b0d93e95437e65bdfce7344653ccd4c1b5a595a6946df402085760"}
Jul 24 21:55:36.257328 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.257304    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:55:36.281461 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281432    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-rbac-proxy-sar-config\") pod \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") "
Jul 24 21:55:36.281576 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281479    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnm2x\" (UniqueName: \"kubernetes.io/projected/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-api-access-pnm2x\") pod \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") "
Jul 24 21:55:36.281576 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281501    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-proxy-tls\") pod \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") "
Jul 24 21:55:36.281576 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281534    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kserve-provision-location\") pod \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") "
Jul 24 21:55:36.281576 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281564    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-cabundle-cert\") pod \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\" (UID: \"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1\") "
Jul 24 21:55:36.281918 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281888    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" (UID: "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:55:36.282008 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.281952    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" (UID: "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:55:36.282049 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.282002    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" (UID: "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:55:36.283764 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.283744    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-api-access-pnm2x" (OuterVolumeSpecName: "kube-api-access-pnm2x") pod "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" (UID: "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1"). InnerVolumeSpecName "kube-api-access-pnm2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:55:36.283764 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.283749    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" (UID: "23b31cb0-ad5b-42c8-b22a-480ba6b0efe1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:55:36.382695 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.382655    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnm2x\" (UniqueName: \"kubernetes.io/projected/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-api-access-pnm2x\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:55:36.382695 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.382694    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:55:36.382943 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.382703    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:55:36.382943 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.382712    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:55:36.382943 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.382721    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:55:36.697478 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.697385    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:36.794704 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794630    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"]
Jul 24 21:55:36.794963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794939    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container"
Jul 24 21:55:36.794963 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794957    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container"
Jul 24 21:55:36.795077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794972    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="storage-initializer"
Jul 24 21:55:36.795077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794978    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="storage-initializer"
Jul 24 21:55:36.795077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794989    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kube-rbac-proxy"
Jul 24 21:55:36.795077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.794995    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kube-rbac-proxy"
Jul 24 21:55:36.795077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.795050    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kserve-container"
Jul 24 21:55:36.795077 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.795060    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" containerName="kube-rbac-proxy"
Jul 24 21:55:36.798185 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.798168    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.801321 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.801292    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 24 21:55:36.801758 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.801740    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 24 21:55:36.812570 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.812531    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"]
Jul 24 21:55:36.885560 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.885528    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.885723 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.885570    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d63145ac-c251-4f53-ac40-073b918c3435-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.885723 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.885608    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d63145ac-c251-4f53-ac40-073b918c3435-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.885723 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.885643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltxh\" (UniqueName: \"kubernetes.io/projected/d63145ac-c251-4f53-ac40-073b918c3435-kube-api-access-7ltxh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.885723 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.885696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.900646 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.900621    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:36.986692 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.986613    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d63145ac-c251-4f53-ac40-073b918c3435-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.986692 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.986645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltxh\" (UniqueName: \"kubernetes.io/projected/d63145ac-c251-4f53-ac40-073b918c3435-kube-api-access-7ltxh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.986692 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.986685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.986905 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.986727    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.986905 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.986757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d63145ac-c251-4f53-ac40-073b918c3435-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.987054 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.987034    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d63145ac-c251-4f53-ac40-073b918c3435-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.987289 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.987275    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.987385 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.987370    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:36.989328 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:36.989311    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d63145ac-c251-4f53-ac40-073b918c3435-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:37.002636 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.002616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltxh\" (UniqueName: \"kubernetes.io/projected/d63145ac-c251-4f53-ac40-073b918c3435-kube-api-access-7ltxh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:37.108860 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.108824    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:55:37.160321 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.160041    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm" event={"ID":"23b31cb0-ad5b-42c8-b22a-480ba6b0efe1","Type":"ContainerDied","Data":"314fa16fb7ffe48ee2e0877007c0f77d1b37de4273991496329fafdfa5f6c3dd"}
Jul 24 21:55:37.160321 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.160122    2585 scope.go:117] "RemoveContainer" containerID="2b843b9610f3db1d5e53d32c7eee2668b294738172c0c24ce83054358e16d828"
Jul 24 21:55:37.160321 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.160301    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"
Jul 24 21:55:37.174445 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.174422    2585 scope.go:117] "RemoveContainer" containerID="2031595610b0d93e95437e65bdfce7344653ccd4c1b5a595a6946df402085760"
Jul 24 21:55:37.182653 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.182634    2585 scope.go:117] "RemoveContainer" containerID="9f9d92840cc933fcaf2c978b4b503710dd0d849bbaf31b4baf209525b3b069c3"
Jul 24 21:55:37.201301 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.201273    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"]
Jul 24 21:55:37.204443 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.204415    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-78dtm"]
Jul 24 21:55:37.250121 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.250101    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"]
Jul 24 21:55:37.252144 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:55:37.252119    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63145ac_c251_4f53_ac40_073b918c3435.slice/crio-84e30d5496f7235311f4225af63b6408c6296a2df4c7a252e1663cc57c014dd4 WatchSource:0}: Error finding container 84e30d5496f7235311f4225af63b6408c6296a2df4c7a252e1663cc57c014dd4: Status 404 returned error can't find the container with id 84e30d5496f7235311f4225af63b6408c6296a2df4c7a252e1663cc57c014dd4
Jul 24 21:55:37.330890 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:37.328995    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b31cb0-ad5b-42c8-b22a-480ba6b0efe1" path="/var/lib/kubelet/pods/23b31cb0-ad5b-42c8-b22a-480ba6b0efe1/volumes"
Jul 24 21:55:38.165227 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:38.165183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerStarted","Data":"81eac2e0dacdf1c4e41cc63668068b39f028ba14e8acde03b23aa4304aa0ec0c"}
Jul 24 21:55:38.165709 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:38.165271    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerStarted","Data":"84e30d5496f7235311f4225af63b6408c6296a2df4c7a252e1663cc57c014dd4"}
Jul 24 21:55:39.531367 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:39.531334    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:55:42.180717 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:42.180685    2585 generic.go:356] "Generic (PLEG): container finished" podID="d63145ac-c251-4f53-ac40-073b918c3435" containerID="81eac2e0dacdf1c4e41cc63668068b39f028ba14e8acde03b23aa4304aa0ec0c" exitCode=0
Jul 24 21:55:42.181062 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:55:42.180760    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerDied","Data":"81eac2e0dacdf1c4e41cc63668068b39f028ba14e8acde03b23aa4304aa0ec0c"}
Jul 24 21:56:00.570508 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:56:00.570466    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:57:42.690240 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:57:42.690212    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerStarted","Data":"af3143de223b68b5aedfc70bebbf319d37257a8eb657e02ede473ffaa8aa26cf"}
Jul 24 21:57:43.696300 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:57:43.696265    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerStarted","Data":"ee9846e2396652e07da063bd33f6fd525da08b5e7676db5cff7b7d0daff1ff9f"}
Jul 24 21:57:43.696815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:57:43.696505    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:57:43.696815 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:57:43.696549    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:57:43.741788 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:57:43.741729    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" podStartSLOduration=3.413291505 podStartE2EDuration="2m7.741709359s" totalImagesPullingTime="2m0.321054676s" totalInitContainerRuntime="4.007363178s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:57:43.739803407 +0000 UTC m=+2544.932981354" watchObservedRunningTime="2026-07-24 21:57:43.741709359 +0000 UTC m=+2544.934887297"
Jul 24 21:57:49.706439 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:57:49.706407    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:58:19.710413 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:19.710380    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:58:26.977436 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:26.977401    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"]
Jul 24 21:58:26.977942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:26.977615    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kserve-container" containerID="cri-o://af3143de223b68b5aedfc70bebbf319d37257a8eb657e02ede473ffaa8aa26cf" gracePeriod=30
Jul 24 21:58:26.977942 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:26.977712    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kube-rbac-proxy" containerID="cri-o://ee9846e2396652e07da063bd33f6fd525da08b5e7676db5cff7b7d0daff1ff9f" gracePeriod=30
Jul 24 21:58:27.018002 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:27.017948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:58:27.863702 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:27.863637    2585 generic.go:356] "Generic (PLEG): container finished" podID="d63145ac-c251-4f53-ac40-073b918c3435" containerID="ee9846e2396652e07da063bd33f6fd525da08b5e7676db5cff7b7d0daff1ff9f" exitCode=2
Jul 24 21:58:27.863702 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:27.863702    2585 generic.go:356] "Generic (PLEG): container finished" podID="d63145ac-c251-4f53-ac40-073b918c3435" containerID="af3143de223b68b5aedfc70bebbf319d37257a8eb657e02ede473ffaa8aa26cf" exitCode=0
Jul 24 21:58:27.863877 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:27.863704    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerDied","Data":"ee9846e2396652e07da063bd33f6fd525da08b5e7676db5cff7b7d0daff1ff9f"}
Jul 24 21:58:27.863877 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:27.863767    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerDied","Data":"af3143de223b68b5aedfc70bebbf319d37257a8eb657e02ede473ffaa8aa26cf"}
Jul 24 21:58:27.911698 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:27.911658    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:58:28.042918 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.042851    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d63145ac-c251-4f53-ac40-073b918c3435-kserve-provision-location\") pod \"d63145ac-c251-4f53-ac40-073b918c3435\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") "
Jul 24 21:58:28.042918 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.042896    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d63145ac-c251-4f53-ac40-073b918c3435-proxy-tls\") pod \"d63145ac-c251-4f53-ac40-073b918c3435\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") "
Jul 24 21:58:28.043356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.042964    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-kube-rbac-proxy-sar-config\") pod \"d63145ac-c251-4f53-ac40-073b918c3435\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") "
Jul 24 21:58:28.043356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.042999    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ltxh\" (UniqueName: \"kubernetes.io/projected/d63145ac-c251-4f53-ac40-073b918c3435-kube-api-access-7ltxh\") pod \"d63145ac-c251-4f53-ac40-073b918c3435\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") "
Jul 24 21:58:28.043356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.043041    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-cabundle-cert\") pod \"d63145ac-c251-4f53-ac40-073b918c3435\" (UID: \"d63145ac-c251-4f53-ac40-073b918c3435\") "
Jul 24 21:58:28.043356 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.043211    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63145ac-c251-4f53-ac40-073b918c3435-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d63145ac-c251-4f53-ac40-073b918c3435" (UID: "d63145ac-c251-4f53-ac40-073b918c3435"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:58:28.043587 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.043388    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d63145ac-c251-4f53-ac40-073b918c3435" (UID: "d63145ac-c251-4f53-ac40-073b918c3435"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:58:28.043587 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.043422    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d63145ac-c251-4f53-ac40-073b918c3435" (UID: "d63145ac-c251-4f53-ac40-073b918c3435"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:58:28.045057 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.045032    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63145ac-c251-4f53-ac40-073b918c3435-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d63145ac-c251-4f53-ac40-073b918c3435" (UID: "d63145ac-c251-4f53-ac40-073b918c3435"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:58:28.045176 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.045121    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63145ac-c251-4f53-ac40-073b918c3435-kube-api-access-7ltxh" (OuterVolumeSpecName: "kube-api-access-7ltxh") pod "d63145ac-c251-4f53-ac40-073b918c3435" (UID: "d63145ac-c251-4f53-ac40-073b918c3435"). InnerVolumeSpecName "kube-api-access-7ltxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:58:28.143538 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.143516    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:58:28.143538 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.143535    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltxh\" (UniqueName: \"kubernetes.io/projected/d63145ac-c251-4f53-ac40-073b918c3435-kube-api-access-7ltxh\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:58:28.143701 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.143545    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d63145ac-c251-4f53-ac40-073b918c3435-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:58:28.143701 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.143554    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d63145ac-c251-4f53-ac40-073b918c3435-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:58:28.143701 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.143562    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d63145ac-c251-4f53-ac40-073b918c3435-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:58:28.869420 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.869394    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"
Jul 24 21:58:28.869587 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.869390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn" event={"ID":"d63145ac-c251-4f53-ac40-073b918c3435","Type":"ContainerDied","Data":"84e30d5496f7235311f4225af63b6408c6296a2df4c7a252e1663cc57c014dd4"}
Jul 24 21:58:28.869587 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.869558    2585 scope.go:117] "RemoveContainer" containerID="ee9846e2396652e07da063bd33f6fd525da08b5e7676db5cff7b7d0daff1ff9f"
Jul 24 21:58:28.878121 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.878104    2585 scope.go:117] "RemoveContainer" containerID="af3143de223b68b5aedfc70bebbf319d37257a8eb657e02ede473ffaa8aa26cf"
Jul 24 21:58:28.885110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.885092    2585 scope.go:117] "RemoveContainer" containerID="81eac2e0dacdf1c4e41cc63668068b39f028ba14e8acde03b23aa4304aa0ec0c"
Jul 24 21:58:28.891309 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.891288    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"]
Jul 24 21:58:28.895051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:28.895032    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-bhhsn"]
Jul 24 21:58:29.326313 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:29.326226    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d63145ac-c251-4f53-ac40-073b918c3435" path="/var/lib/kubelet/pods/d63145ac-c251-4f53-ac40-073b918c3435/volumes"
Jul 24 21:58:32.025357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.025311    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:58:32.084742 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.084716    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"]
Jul 24 21:58:32.085042 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085025    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="storage-initializer"
Jul 24 21:58:32.085122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085046    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="storage-initializer"
Jul 24 21:58:32.085122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085065    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kserve-container"
Jul 24 21:58:32.085122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085073    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kserve-container"
Jul 24 21:58:32.085122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085082    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kube-rbac-proxy"
Jul 24 21:58:32.085122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085091    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kube-rbac-proxy"
Jul 24 21:58:32.085389 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085175    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kube-rbac-proxy"
Jul 24 21:58:32.085389 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.085186    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d63145ac-c251-4f53-ac40-073b918c3435" containerName="kserve-container"
Jul 24 21:58:32.088136 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.088116    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.090845 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.090826    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:58:32.090977 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.090958    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 24 21:58:32.091110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.090967    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:58:32.091110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.091018    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:58:32.091110 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.091048    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:58:32.091242 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.091212    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 24 21:58:32.091242 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.091222    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:58:32.097530 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.097510    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"]
Jul 24 21:58:32.149451 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.149420    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:58:32.173114 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.173086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.173231 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.173120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa761ef8-3bd1-4cc9-a160-11989fb00469-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.173231 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.173158    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa761ef8-3bd1-4cc9-a160-11989fb00469-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.173231 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.173192    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.173365 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.173235    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8p5g\" (UniqueName: \"kubernetes.io/projected/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-api-access-b8p5g\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274019 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.273993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8p5g\" (UniqueName: \"kubernetes.io/projected/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-api-access-b8p5g\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274028    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa761ef8-3bd1-4cc9-a160-11989fb00469-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa761ef8-3bd1-4cc9-a160-11989fb00469-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274623    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa761ef8-3bd1-4cc9-a160-11989fb00469-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274820 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.274882 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.274850    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.276648 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.276603    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa761ef8-3bd1-4cc9-a160-11989fb00469-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.283846 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.283822    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8p5g\" (UniqueName: \"kubernetes.io/projected/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-api-access-b8p5g\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.399741 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.399713    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:32.527071 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.527048    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"]
Jul 24 21:58:32.528963 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:58:32.528933    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa761ef8_3bd1_4cc9_a160_11989fb00469.slice/crio-aae76d6381f295b71574a1bc3d7eb93b3bc870fd395098d4e3469173d6ab9641 WatchSource:0}: Error finding container aae76d6381f295b71574a1bc3d7eb93b3bc870fd395098d4e3469173d6ab9641: Status 404 returned error can't find the container with id aae76d6381f295b71574a1bc3d7eb93b3bc870fd395098d4e3469173d6ab9641
Jul 24 21:58:32.887994 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.887956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerStarted","Data":"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"}
Jul 24 21:58:32.888154 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:32.888017    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerStarted","Data":"aae76d6381f295b71574a1bc3d7eb93b3bc870fd395098d4e3469173d6ab9641"}
Jul 24 21:58:36.904217 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:36.904181    2585 generic.go:356] "Generic (PLEG): container finished" podID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerID="038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452" exitCode=0
Jul 24 21:58:36.904625 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:36.904254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerDied","Data":"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"}
Jul 24 21:58:37.910265 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:37.910228    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerStarted","Data":"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"}
Jul 24 21:58:37.910786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:37.910291    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerStarted","Data":"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"}
Jul 24 21:58:37.910786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:37.910602    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:37.910786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:37.910697    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:37.912226 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:37.912198    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 24 21:58:37.933305 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:37.933253    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" podStartSLOduration=2.308189131 podStartE2EDuration="5.93324051s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.625051379s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:58:37.931650705 +0000 UTC m=+2599.124828677" watchObservedRunningTime="2026-07-24 21:58:37.93324051 +0000 UTC m=+2599.126418445"
Jul 24 21:58:38.914903 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:38.914864    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 24 21:58:43.919266 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:43.919184    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:58:43.919750 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:43.919700    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 24 21:58:53.920094 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:58:53.920055    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:59:02.114265 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.114229    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"]
Jul 24 21:59:02.114901 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.114539    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container" containerID="cri-o://f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6" gracePeriod=30
Jul 24 21:59:02.114901 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.114545    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kube-rbac-proxy" containerID="cri-o://f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1" gracePeriod=30
Jul 24 21:59:02.146602 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.146577    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:59:02.753327 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.753305    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:59:02.901909 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.901883    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-rbac-proxy-sar-config\") pod \"fa761ef8-3bd1-4cc9-a160-11989fb00469\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") "
Jul 24 21:59:02.902051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.901946    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa761ef8-3bd1-4cc9-a160-11989fb00469-proxy-tls\") pod \"fa761ef8-3bd1-4cc9-a160-11989fb00469\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") "
Jul 24 21:59:02.902051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.901991    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-cabundle-cert\") pod \"fa761ef8-3bd1-4cc9-a160-11989fb00469\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") "
Jul 24 21:59:02.902051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.902007    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8p5g\" (UniqueName: \"kubernetes.io/projected/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-api-access-b8p5g\") pod \"fa761ef8-3bd1-4cc9-a160-11989fb00469\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") "
Jul 24 21:59:02.902051 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.902028    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa761ef8-3bd1-4cc9-a160-11989fb00469-kserve-provision-location\") pod \"fa761ef8-3bd1-4cc9-a160-11989fb00469\" (UID: \"fa761ef8-3bd1-4cc9-a160-11989fb00469\") "
Jul 24 21:59:02.902344 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.902321    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa761ef8-3bd1-4cc9-a160-11989fb00469" (UID: "fa761ef8-3bd1-4cc9-a160-11989fb00469"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:59:02.902406 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.902358    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa761ef8-3bd1-4cc9-a160-11989fb00469-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa761ef8-3bd1-4cc9-a160-11989fb00469" (UID: "fa761ef8-3bd1-4cc9-a160-11989fb00469"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:59:02.902444 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.902415    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa761ef8-3bd1-4cc9-a160-11989fb00469" (UID: "fa761ef8-3bd1-4cc9-a160-11989fb00469"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:59:02.904291 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.904265    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-api-access-b8p5g" (OuterVolumeSpecName: "kube-api-access-b8p5g") pod "fa761ef8-3bd1-4cc9-a160-11989fb00469" (UID: "fa761ef8-3bd1-4cc9-a160-11989fb00469"). InnerVolumeSpecName "kube-api-access-b8p5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:59:02.904377 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:02.904285    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa761ef8-3bd1-4cc9-a160-11989fb00469-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa761ef8-3bd1-4cc9-a160-11989fb00469" (UID: "fa761ef8-3bd1-4cc9-a160-11989fb00469"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:59:03.002416 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.002388    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa761ef8-3bd1-4cc9-a160-11989fb00469-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:03.002527 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.002419    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:03.002527 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.002431    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8p5g\" (UniqueName: \"kubernetes.io/projected/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-api-access-b8p5g\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:03.002527 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.002440    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa761ef8-3bd1-4cc9-a160-11989fb00469-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:03.002527 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.002452    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa761ef8-3bd1-4cc9-a160-11989fb00469-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:03.004156 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004131    2585 generic.go:356] "Generic (PLEG): container finished" podID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerID="f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1" exitCode=2
Jul 24 21:59:03.004156 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004155    2585 generic.go:356] "Generic (PLEG): container finished" podID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerID="f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6" exitCode=0
Jul 24 21:59:03.004287 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004199    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"
Jul 24 21:59:03.004287 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerDied","Data":"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"}
Jul 24 21:59:03.004287 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004264    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerDied","Data":"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"}
Jul 24 21:59:03.004392 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004295    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968" event={"ID":"fa761ef8-3bd1-4cc9-a160-11989fb00469","Type":"ContainerDied","Data":"aae76d6381f295b71574a1bc3d7eb93b3bc870fd395098d4e3469173d6ab9641"}
Jul 24 21:59:03.004392 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.004340    2585 scope.go:117] "RemoveContainer" containerID="f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"
Jul 24 21:59:03.013369 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.013353    2585 scope.go:117] "RemoveContainer" containerID="f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"
Jul 24 21:59:03.020733 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.020709    2585 scope.go:117] "RemoveContainer" containerID="038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"
Jul 24 21:59:03.027316 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.027296    2585 scope.go:117] "RemoveContainer" containerID="f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"
Jul 24 21:59:03.027574 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:59:03.027555    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1\": container with ID starting with f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1 not found: ID does not exist" containerID="f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"
Jul 24 21:59:03.027642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.027586    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"} err="failed to get container status \"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1\": rpc error: code = NotFound desc = could not find container \"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1\": container with ID starting with f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1 not found: ID does not exist"
Jul 24 21:59:03.027642 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.027610    2585 scope.go:117] "RemoveContainer" containerID="f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"
Jul 24 21:59:03.027891 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:59:03.027873    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6\": container with ID starting with f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6 not found: ID does not exist" containerID="f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"
Jul 24 21:59:03.027961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.027896    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"} err="failed to get container status \"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6\": rpc error: code = NotFound desc = could not find container \"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6\": container with ID starting with f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6 not found: ID does not exist"
Jul 24 21:59:03.027961 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.027913    2585 scope.go:117] "RemoveContainer" containerID="038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"
Jul 24 21:59:03.028142 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:59:03.028125    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452\": container with ID starting with 038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452 not found: ID does not exist" containerID="038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"
Jul 24 21:59:03.028178 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028147    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"} err="failed to get container status \"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452\": rpc error: code = NotFound desc = could not find container \"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452\": container with ID starting with 038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452 not found: ID does not exist"
Jul 24 21:59:03.028178 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028161    2585 scope.go:117] "RemoveContainer" containerID="f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"
Jul 24 21:59:03.028359 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028344    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1"} err="failed to get container status \"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1\": rpc error: code = NotFound desc = could not find container \"f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1\": container with ID starting with f403391748f04dad0cb4a7e92361f7e8ebcebd686ccbbeb014b9db77fbc72ff1 not found: ID does not exist"
Jul 24 21:59:03.028399 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028361    2585 scope.go:117] "RemoveContainer" containerID="f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"
Jul 24 21:59:03.028533 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028517    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6"} err="failed to get container status \"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6\": rpc error: code = NotFound desc = could not find container \"f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6\": container with ID starting with f681926c774585cda983359d611ab76530c727dc8b70eb4e07e08285487a08a6 not found: ID does not exist"
Jul 24 21:59:03.028584 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028534    2585 scope.go:117] "RemoveContainer" containerID="038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"
Jul 24 21:59:03.028822 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.028801    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452"} err="failed to get container status \"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452\": rpc error: code = NotFound desc = could not find container \"038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452\": container with ID starting with 038f9cb4f2bc8a023bf56722bfb82a17f4aa73955335e7147bb543eeaf361452 not found: ID does not exist"
Jul 24 21:59:03.044523 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.044497    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"]
Jul 24 21:59:03.056736 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.056716    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cp968"]
Jul 24 21:59:03.326779 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:03.326710    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" path="/var/lib/kubelet/pods/fa761ef8-3bd1-4cc9-a160-11989fb00469/volumes"
Jul 24 21:59:07.270235 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.270186    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:59:07.297131 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297095    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"]
Jul 24 21:59:07.297416 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297403    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container"
Jul 24 21:59:07.297478 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297419    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container"
Jul 24 21:59:07.297478 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297428    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="storage-initializer"
Jul 24 21:59:07.297478 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297433    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="storage-initializer"
Jul 24 21:59:07.297478 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297449    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kube-rbac-proxy"
Jul 24 21:59:07.297478 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297457    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kube-rbac-proxy"
Jul 24 21:59:07.297712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297514    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kserve-container"
Jul 24 21:59:07.297712 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.297528    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa761ef8-3bd1-4cc9-a160-11989fb00469" containerName="kube-rbac-proxy"
Jul 24 21:59:07.300588 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.300570    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.303199 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.303171    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 24 21:59:07.303311 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.303215    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 24 21:59:07.303419 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.303401    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:59:07.303690 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.303656    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:59:07.304462 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.304443    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 21:59:07.304574 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.304489    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:59:07.304574 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.304539    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:59:07.311308 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.311288    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"]
Jul 24 21:59:07.390255 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.390218    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:59:07.430768 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.430739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.430893 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.430776    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.430893 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.430806    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13ef07a-911f-4cec-87c1-a74d604147e5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.430893 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.430853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13ef07a-911f-4cec-87c1-a74d604147e5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.430893 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.430876    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rj6d\" (UniqueName: \"kubernetes.io/projected/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-api-access-5rj6d\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.531953    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13ef07a-911f-4cec-87c1-a74d604147e5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532014 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.531986    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rj6d\" (UniqueName: \"kubernetes.io/projected/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-api-access-5rj6d\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.532029    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.532047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532122 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.532074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13ef07a-911f-4cec-87c1-a74d604147e5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532508 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.532471    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13ef07a-911f-4cec-87c1-a74d604147e5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532814 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.532794    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.532885 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.532820    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.534735 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.534715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13ef07a-911f-4cec-87c1-a74d604147e5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.541349 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.541320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rj6d\" (UniqueName: \"kubernetes.io/projected/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-api-access-5rj6d\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.612950 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.612916    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:07.742223 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:07.742199    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"]
Jul 24 21:59:07.744100 ip-10-0-143-243 kubenswrapper[2585]: W0724 21:59:07.744069    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13ef07a_911f_4cec_87c1_a74d604147e5.slice/crio-e58cca90ba7cac0e3c3f01bf7961ae5480892aa55d6e3bfdf963abd06abe37c5 WatchSource:0}: Error finding container e58cca90ba7cac0e3c3f01bf7961ae5480892aa55d6e3bfdf963abd06abe37c5: Status 404 returned error can't find the container with id e58cca90ba7cac0e3c3f01bf7961ae5480892aa55d6e3bfdf963abd06abe37c5
Jul 24 21:59:08.023553 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:08.023516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerStarted","Data":"006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2"}
Jul 24 21:59:08.023713 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:08.023572    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerStarted","Data":"e58cca90ba7cac0e3c3f01bf7961ae5480892aa55d6e3bfdf963abd06abe37c5"}
Jul 24 21:59:12.040786 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:12.040748    2585 generic.go:356] "Generic (PLEG): container finished" podID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerID="006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2" exitCode=0
Jul 24 21:59:12.041166 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:12.040822    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerDied","Data":"006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2"}
Jul 24 21:59:13.047493 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:13.047449    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerStarted","Data":"e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4"}
Jul 24 21:59:13.047933 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:13.047542    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerStarted","Data":"b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54"}
Jul 24 21:59:13.069600 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:13.069542    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" podStartSLOduration=2.008617152 podStartE2EDuration="6.069523341s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.060906189s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:59:13.068951524 +0000 UTC m=+2634.262129476" watchObservedRunningTime="2026-07-24 21:59:13.069523341 +0000 UTC m=+2634.262701279"
Jul 24 21:59:18.048595 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:18.048557    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:18.049081 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:18.049052    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:18.053681 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:18.053644    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:49.074095 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:49.074065    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:57.424287 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:57.424251    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"]
Jul 24 21:59:57.424877 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:57.424556    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kserve-container" containerID="cri-o://b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54" gracePeriod=30
Jul 24 21:59:57.424877 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:57.424623    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kube-rbac-proxy" containerID="cri-o://e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4" gracePeriod=30
Jul 24 21:59:57.448955 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:57.448928    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 21:59:58.054549 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.054509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 24 21:59:58.214924 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.214888    2585 generic.go:356] "Generic (PLEG): container finished" podID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerID="e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4" exitCode=2
Jul 24 21:59:58.215095 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.214955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerDied","Data":"e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4"}
Jul 24 21:59:58.567220 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.567198    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:58.678133 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678070    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13ef07a-911f-4cec-87c1-a74d604147e5-proxy-tls\") pod \"d13ef07a-911f-4cec-87c1-a74d604147e5\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") "
Jul 24 21:59:58.678133 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678129    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-cabundle-cert\") pod \"d13ef07a-911f-4cec-87c1-a74d604147e5\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") "
Jul 24 21:59:58.678319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678151    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13ef07a-911f-4cec-87c1-a74d604147e5-kserve-provision-location\") pod \"d13ef07a-911f-4cec-87c1-a74d604147e5\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") "
Jul 24 21:59:58.678319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678177    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-rbac-proxy-sar-config\") pod \"d13ef07a-911f-4cec-87c1-a74d604147e5\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") "
Jul 24 21:59:58.678319 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678196    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rj6d\" (UniqueName: \"kubernetes.io/projected/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-api-access-5rj6d\") pod \"d13ef07a-911f-4cec-87c1-a74d604147e5\" (UID: \"d13ef07a-911f-4cec-87c1-a74d604147e5\") "
Jul 24 21:59:58.678558 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678523    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13ef07a-911f-4cec-87c1-a74d604147e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d13ef07a-911f-4cec-87c1-a74d604147e5" (UID: "d13ef07a-911f-4cec-87c1-a74d604147e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:59:58.678558 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678549    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d13ef07a-911f-4cec-87c1-a74d604147e5" (UID: "d13ef07a-911f-4cec-87c1-a74d604147e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:59:58.678779 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.678526    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d13ef07a-911f-4cec-87c1-a74d604147e5" (UID: "d13ef07a-911f-4cec-87c1-a74d604147e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:59:58.680261 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.680242    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13ef07a-911f-4cec-87c1-a74d604147e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d13ef07a-911f-4cec-87c1-a74d604147e5" (UID: "d13ef07a-911f-4cec-87c1-a74d604147e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:59:58.680398 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.680375    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-api-access-5rj6d" (OuterVolumeSpecName: "kube-api-access-5rj6d") pod "d13ef07a-911f-4cec-87c1-a74d604147e5" (UID: "d13ef07a-911f-4cec-87c1-a74d604147e5"). InnerVolumeSpecName "kube-api-access-5rj6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:59:58.779002 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.778967    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:58.779002 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.778997    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rj6d\" (UniqueName: \"kubernetes.io/projected/d13ef07a-911f-4cec-87c1-a74d604147e5-kube-api-access-5rj6d\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:58.779002 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.779008    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13ef07a-911f-4cec-87c1-a74d604147e5-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:58.779189 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.779018    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d13ef07a-911f-4cec-87c1-a74d604147e5-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:58.779189 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:58.779025    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13ef07a-911f-4cec-87c1-a74d604147e5-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 21:59:59.220629 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.220590    2585 generic.go:356] "Generic (PLEG): container finished" podID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerID="b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54" exitCode=0
Jul 24 21:59:59.220809 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.220659    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"
Jul 24 21:59:59.220809 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.220690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerDied","Data":"b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54"}
Jul 24 21:59:59.220809 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.220756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl" event={"ID":"d13ef07a-911f-4cec-87c1-a74d604147e5","Type":"ContainerDied","Data":"e58cca90ba7cac0e3c3f01bf7961ae5480892aa55d6e3bfdf963abd06abe37c5"}
Jul 24 21:59:59.220809 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.220783    2585 scope.go:117] "RemoveContainer" containerID="e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4"
Jul 24 21:59:59.229525 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.229510    2585 scope.go:117] "RemoveContainer" containerID="b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54"
Jul 24 21:59:59.242820 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.242796    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"]
Jul 24 21:59:59.242891 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.242844    2585 scope.go:117] "RemoveContainer" containerID="006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2"
Jul 24 21:59:59.246550 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.246523    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-56bxl"]
Jul 24 21:59:59.250083 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.250050    2585 scope.go:117] "RemoveContainer" containerID="e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4"
Jul 24 21:59:59.250298 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:59:59.250281    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4\": container with ID starting with e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4 not found: ID does not exist" containerID="e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4"
Jul 24 21:59:59.250357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.250305    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4"} err="failed to get container status \"e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4\": rpc error: code = NotFound desc = could not find container \"e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4\": container with ID starting with e6114ccde4ae7fe45b3d8af9b05a6e4f1c767c14e2419d0093fd4f3573f35ee4 not found: ID does not exist"
Jul 24 21:59:59.250357 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.250320    2585 scope.go:117] "RemoveContainer" containerID="b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54"
Jul 24 21:59:59.250550 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:59:59.250531    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54\": container with ID starting with b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54 not found: ID does not exist" containerID="b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54"
Jul 24 21:59:59.250594 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.250557    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54"} err="failed to get container status \"b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54\": rpc error: code = NotFound desc = could not find container \"b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54\": container with ID starting with b45c7a6979bcd44bd28b215d28b1a73a9c0729436ebcda860357b1e56183fe54 not found: ID does not exist"
Jul 24 21:59:59.250594 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.250571    2585 scope.go:117] "RemoveContainer" containerID="006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2"
Jul 24 21:59:59.250842 ip-10-0-143-243 kubenswrapper[2585]: E0724 21:59:59.250827    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2\": container with ID starting with 006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2 not found: ID does not exist" containerID="006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2"
Jul 24 21:59:59.250904 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.250845    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2"} err="failed to get container status \"006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2\": rpc error: code = NotFound desc = could not find container \"006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2\": container with ID starting with 006d003df63aee718ce7a0e9148e296287c110ea1e602209f00b74cc17527ac2 not found: ID does not exist"
Jul 24 21:59:59.326551 ip-10-0-143-243 kubenswrapper[2585]: I0724 21:59:59.326527    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" path="/var/lib/kubelet/pods/d13ef07a-911f-4cec-87c1-a74d604147e5/volumes"
Jul 24 22:00:02.531878 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.531844    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:00:02.573618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573595    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"]
Jul 24 22:00:02.573881 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573868    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kserve-container"
Jul 24 22:00:02.573926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573882    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kserve-container"
Jul 24 22:00:02.573926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573894    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kube-rbac-proxy"
Jul 24 22:00:02.573926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573899    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kube-rbac-proxy"
Jul 24 22:00:02.573926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573907    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="storage-initializer"
Jul 24 22:00:02.573926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573912    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="storage-initializer"
Jul 24 22:00:02.574075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573964    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kube-rbac-proxy"
Jul 24 22:00:02.574075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.573971    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d13ef07a-911f-4cec-87c1-a74d604147e5" containerName="kserve-container"
Jul 24 22:00:02.576949 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.576934    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.579726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.579636    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:00:02.579726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.579650    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 24 22:00:02.579726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.579711    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:00:02.579726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.579718    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:00:02.579975 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.579781    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:00:02.580036 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.580019    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 24 22:00:02.580408 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.580389    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:00:02.590959 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.590941    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"]
Jul 24 22:00:02.603746 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.603719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.603877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.603762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwgd\" (UniqueName: \"kubernetes.io/projected/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-api-access-njwgd\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.603877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.603802    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae015377-9f79-439f-9d6b-1e18fcf58a50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.603877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.603861    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae015377-9f79-439f-9d6b-1e18fcf58a50-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.604017 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.603908    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.704969 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.704943    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.704969 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.704973    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwgd\" (UniqueName: \"kubernetes.io/projected/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-api-access-njwgd\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.705213 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.704996    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae015377-9f79-439f-9d6b-1e18fcf58a50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.705213 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.705070    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae015377-9f79-439f-9d6b-1e18fcf58a50-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.705213 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.705107    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.705567 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.705538    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae015377-9f79-439f-9d6b-1e18fcf58a50-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.705804 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.705783    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.705876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.705804    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.707772 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.707748    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae015377-9f79-439f-9d6b-1e18fcf58a50-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.713568 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.713550    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwgd\" (UniqueName: \"kubernetes.io/projected/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-api-access-njwgd\") pod \"isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:02.721612 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.721587    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:00:02.888501 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:02.888476    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:03.019810 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:03.019746    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"]
Jul 24 22:00:03.022349 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:00:03.022320    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae015377_9f79_439f_9d6b_1e18fcf58a50.slice/crio-6bbb170750337b85f5a10e2f1d80c50d6595b84929e0742216207e5fcddb01b7 WatchSource:0}: Error finding container 6bbb170750337b85f5a10e2f1d80c50d6595b84929e0742216207e5fcddb01b7: Status 404 returned error can't find the container with id 6bbb170750337b85f5a10e2f1d80c50d6595b84929e0742216207e5fcddb01b7
Jul 24 22:00:03.237217 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:03.237132    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerStarted","Data":"37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d"}
Jul 24 22:00:03.237217 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:03.237203    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerStarted","Data":"6bbb170750337b85f5a10e2f1d80c50d6595b84929e0742216207e5fcddb01b7"}
Jul 24 22:00:07.253207 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:07.253170    2585 generic.go:356] "Generic (PLEG): container finished" podID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerID="37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d" exitCode=0
Jul 24 22:00:07.253626 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:07.253214    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerDied","Data":"37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d"}
Jul 24 22:00:07.312876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:07.312859    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:00:08.258928 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:08.258897    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerStarted","Data":"3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8"}
Jul 24 22:00:10.269303 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:10.269221    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerStarted","Data":"5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4"}
Jul 24 22:00:10.269303 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:10.269283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerStarted","Data":"f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307"}
Jul 24 22:00:10.269764 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:10.269329    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:10.269764 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:10.269457    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:10.269764 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:10.269522    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:10.292247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:10.292194    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podStartSLOduration=1.926869607 podStartE2EDuration="8.292174303s" totalImagesPullingTime="2.635445749s" totalInitContainerRuntime="3.729858947s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:00:10.291154289 +0000 UTC m=+2691.484332254" watchObservedRunningTime="2026-07-24 22:00:10.292174303 +0000 UTC m=+2691.485352241"
Jul 24 22:00:16.279199 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:16.279164    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:00:46.282040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:00:46.282004    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:01:16.282970 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:16.282940    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:01:22.664530 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:22.664477    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"]
Jul 24 22:01:22.664983 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:22.664827    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container" containerID="cri-o://3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8" gracePeriod=30
Jul 24 22:01:22.664983 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:22.664928    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" containerID="cri-o://5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4" gracePeriod=30
Jul 24 22:01:22.665135 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:22.664928    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-agent" containerID="cri-o://f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307" gracePeriod=30
Jul 24 22:01:22.699277 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:22.699243    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:01:23.534334 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:23.534303    2585 generic.go:356] "Generic (PLEG): container finished" podID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerID="5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4" exitCode=2
Jul 24 22:01:23.534502 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:23.534375    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerDied","Data":"5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4"}
Jul 24 22:01:25.544193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:25.544157    2585 generic.go:356] "Generic (PLEG): container finished" podID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerID="3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8" exitCode=0
Jul 24 22:01:25.544579 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:25.544228    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerDied","Data":"3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8"}
Jul 24 22:01:26.274758 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:26.274715    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 24 22:01:26.279978 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:26.279953    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 24 22:01:27.787144 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:27.787100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:01:27.913076 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:27.913038    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:01:31.274252 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:31.274214    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 24 22:01:36.273846 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:36.273799    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 24 22:01:36.274243 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:36.273985    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:01:36.279852 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:36.279814    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 24 22:01:41.274025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:41.273941    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 24 22:01:46.273987 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:46.273946    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 24 22:01:46.280384 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:46.280358    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 24 22:01:46.280478 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:46.280458    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:01:51.274088 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:51.274048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 24 22:01:52.816168 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.816147    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:01:52.880540 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880518    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-rbac-proxy-sar-config\") pod \"ae015377-9f79-439f-9d6b-1e18fcf58a50\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") "
Jul 24 22:01:52.880704 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880555    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae015377-9f79-439f-9d6b-1e18fcf58a50-kserve-provision-location\") pod \"ae015377-9f79-439f-9d6b-1e18fcf58a50\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") "
Jul 24 22:01:52.880704 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880599    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njwgd\" (UniqueName: \"kubernetes.io/projected/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-api-access-njwgd\") pod \"ae015377-9f79-439f-9d6b-1e18fcf58a50\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") "
Jul 24 22:01:52.880704 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880638    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae015377-9f79-439f-9d6b-1e18fcf58a50-proxy-tls\") pod \"ae015377-9f79-439f-9d6b-1e18fcf58a50\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") "
Jul 24 22:01:52.880704 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880655    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-cabundle-cert\") pod \"ae015377-9f79-439f-9d6b-1e18fcf58a50\" (UID: \"ae015377-9f79-439f-9d6b-1e18fcf58a50\") "
Jul 24 22:01:52.880987 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880923    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae015377-9f79-439f-9d6b-1e18fcf58a50" (UID: "ae015377-9f79-439f-9d6b-1e18fcf58a50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:01:52.881086 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.880928    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae015377-9f79-439f-9d6b-1e18fcf58a50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae015377-9f79-439f-9d6b-1e18fcf58a50" (UID: "ae015377-9f79-439f-9d6b-1e18fcf58a50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:01:52.881086 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.881046    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae015377-9f79-439f-9d6b-1e18fcf58a50" (UID: "ae015377-9f79-439f-9d6b-1e18fcf58a50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:01:52.882904 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.882880    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-api-access-njwgd" (OuterVolumeSpecName: "kube-api-access-njwgd") pod "ae015377-9f79-439f-9d6b-1e18fcf58a50" (UID: "ae015377-9f79-439f-9d6b-1e18fcf58a50"). InnerVolumeSpecName "kube-api-access-njwgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:01:52.883006 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.882983    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae015377-9f79-439f-9d6b-1e18fcf58a50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae015377-9f79-439f-9d6b-1e18fcf58a50" (UID: "ae015377-9f79-439f-9d6b-1e18fcf58a50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:01:52.981935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.981860    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae015377-9f79-439f-9d6b-1e18fcf58a50-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:01:52.981935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.981895    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:01:52.981935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.981905    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:01:52.981935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.981915    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae015377-9f79-439f-9d6b-1e18fcf58a50-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:01:52.981935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:52.981923    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njwgd\" (UniqueName: \"kubernetes.io/projected/ae015377-9f79-439f-9d6b-1e18fcf58a50-kube-api-access-njwgd\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:01:53.650150 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.650114    2585 generic.go:356] "Generic (PLEG): container finished" podID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerID="f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307" exitCode=0
Jul 24 22:01:53.650315 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.650156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerDied","Data":"f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307"}
Jul 24 22:01:53.650315 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.650220    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"
Jul 24 22:01:53.650315 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.650230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv" event={"ID":"ae015377-9f79-439f-9d6b-1e18fcf58a50","Type":"ContainerDied","Data":"6bbb170750337b85f5a10e2f1d80c50d6595b84929e0742216207e5fcddb01b7"}
Jul 24 22:01:53.650315 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.650250    2585 scope.go:117] "RemoveContainer" containerID="5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4"
Jul 24 22:01:53.658381 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.658367    2585 scope.go:117] "RemoveContainer" containerID="f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307"
Jul 24 22:01:53.665544 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.665524    2585 scope.go:117] "RemoveContainer" containerID="3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8"
Jul 24 22:01:53.670537 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.670516    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"]
Jul 24 22:01:53.674121 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.673960    2585 scope.go:117] "RemoveContainer" containerID="37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d"
Jul 24 22:01:53.674356 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.674334    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5cf4b5779c-s95hv"]
Jul 24 22:01:53.680729 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.680712    2585 scope.go:117] "RemoveContainer" containerID="5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4"
Jul 24 22:01:53.680986 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:01:53.680967    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4\": container with ID starting with 5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4 not found: ID does not exist" containerID="5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4"
Jul 24 22:01:53.681033 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.680995    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4"} err="failed to get container status \"5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4\": rpc error: code = NotFound desc = could not find container \"5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4\": container with ID starting with 5e34b97aefe2ad7cba826863383fe1ba7c406cb6f746fd96aef6e1a85cf0b3b4 not found: ID does not exist"
Jul 24 22:01:53.681033 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.681013    2585 scope.go:117] "RemoveContainer" containerID="f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307"
Jul 24 22:01:53.681219 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:01:53.681205    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307\": container with ID starting with f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307 not found: ID does not exist" containerID="f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307"
Jul 24 22:01:53.681260 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.681223    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307"} err="failed to get container status \"f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307\": rpc error: code = NotFound desc = could not find container \"f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307\": container with ID starting with f23916a22a9f77eaa78801e9b8ac68b748c037047c7f55d43cb4a38e00759307 not found: ID does not exist"
Jul 24 22:01:53.681260 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.681235    2585 scope.go:117] "RemoveContainer" containerID="3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8"
Jul 24 22:01:53.681421 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:01:53.681407    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8\": container with ID starting with 3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8 not found: ID does not exist" containerID="3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8"
Jul 24 22:01:53.681460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.681424    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8"} err="failed to get container status \"3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8\": rpc error: code = NotFound desc = could not find container \"3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8\": container with ID starting with 3f54952f143bbdacad3e64698cc7c15e6486ce19fd206dd08091f39049690fe8 not found: ID does not exist"
Jul 24 22:01:53.681460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.681435    2585 scope.go:117] "RemoveContainer" containerID="37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d"
Jul 24 22:01:53.681627 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:01:53.681607    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d\": container with ID starting with 37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d not found: ID does not exist" containerID="37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d"
Jul 24 22:01:53.681682 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:53.681634    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d"} err="failed to get container status \"37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d\": rpc error: code = NotFound desc = could not find container \"37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d\": container with ID starting with 37ceb69d9b6638d0bdf331c3b50287af136826a3afaa403784f0445a5f8ba04d not found: ID does not exist"
Jul 24 22:01:55.326954 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:01:55.326918    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" path="/var/lib/kubelet/pods/ae015377-9f79-439f-9d6b-1e18fcf58a50/volumes"
Jul 24 22:02:39.508463 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:39.508414    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:02:44.657034 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.656994    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:02:44.692876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.692843    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"]
Jul 24 22:02:44.693143 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693130    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="storage-initializer"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693144    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="storage-initializer"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693152    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-agent"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693158    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-agent"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693164    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693169    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693175    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container"
Jul 24 22:02:44.693193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693180    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container"
Jul 24 22:02:44.693401 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693222    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-container"
Jul 24 22:02:44.693401 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693229    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kube-rbac-proxy"
Jul 24 22:02:44.693401 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.693236    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae015377-9f79-439f-9d6b-1e18fcf58a50" containerName="kserve-agent"
Jul 24 22:02:44.695932 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.695917    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.698693 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.698637    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:02:44.698827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.698705    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:02:44.698827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.698724    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:02:44.698827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.698733    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:02:44.698827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.698747    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 24 22:02:44.699094 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.698867    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:02:44.699861 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.699843    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:02:44.707374 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.707352    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"]
Jul 24 22:02:44.845789 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.845754    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.845941 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.845798    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55736fe1-104c-4dca-b699-89ea05ae2863-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.845941 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.845823    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4gm7\" (UniqueName: \"kubernetes.io/projected/55736fe1-104c-4dca-b699-89ea05ae2863-kube-api-access-h4gm7\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.845941 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.845928    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55736fe1-104c-4dca-b699-89ea05ae2863-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.846074 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.845956    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.871380 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.871351    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:02:44.946637 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.946577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55736fe1-104c-4dca-b699-89ea05ae2863-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.946637 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.946608    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4gm7\" (UniqueName: \"kubernetes.io/projected/55736fe1-104c-4dca-b699-89ea05ae2863-kube-api-access-h4gm7\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.946800 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.946741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55736fe1-104c-4dca-b699-89ea05ae2863-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.946800 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.946773    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.946903 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.946814    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.946984 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.946964    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55736fe1-104c-4dca-b699-89ea05ae2863-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.947476 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.947456    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.947561 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.947463    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.949307 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.949282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55736fe1-104c-4dca-b699-89ea05ae2863-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:44.954984 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:44.954961    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4gm7\" (UniqueName: \"kubernetes.io/projected/55736fe1-104c-4dca-b699-89ea05ae2863-kube-api-access-h4gm7\") pod \"isvc-paddle-runtime-predictor-58b99fff76-v9vk8\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:45.006777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:45.006756    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:02:45.136908 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:45.136883    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"]
Jul 24 22:02:45.139184 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:02:45.139159    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55736fe1_104c_4dca_b699_89ea05ae2863.slice/crio-1cff6bfc1a250226fff65b07456ca2c0f5aa3f553fa7903926813e995db046f7 WatchSource:0}: Error finding container 1cff6bfc1a250226fff65b07456ca2c0f5aa3f553fa7903926813e995db046f7: Status 404 returned error can't find the container with id 1cff6bfc1a250226fff65b07456ca2c0f5aa3f553fa7903926813e995db046f7
Jul 24 22:02:45.840102 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:45.840067    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerStarted","Data":"7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169"}
Jul 24 22:02:45.840485 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:45.840120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerStarted","Data":"1cff6bfc1a250226fff65b07456ca2c0f5aa3f553fa7903926813e995db046f7"}
Jul 24 22:02:50.867025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:50.866989    2585 generic.go:356] "Generic (PLEG): container finished" podID="55736fe1-104c-4dca-b699-89ea05ae2863" containerID="7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169" exitCode=0
Jul 24 22:02:50.867460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:02:50.867047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerDied","Data":"7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169"}
Jul 24 22:03:02.920178 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:02.920135    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerStarted","Data":"32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73"}
Jul 24 22:03:02.920585 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:02.920198    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerStarted","Data":"59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9"}
Jul 24 22:03:02.920585 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:02.920476    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:03:02.920688 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:02.920629    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:03:02.922063 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:02.922042    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 24 22:03:02.940281 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:02.940232    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podStartSLOduration=2.454232595 podStartE2EDuration="18.940220515s" totalImagesPullingTime="11.4157158s" totalInitContainerRuntime="5.07027212s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:03:02.939948431 +0000 UTC m=+2864.133126423" watchObservedRunningTime="2026-07-24 22:03:02.940220515 +0000 UTC m=+2864.133398451"
Jul 24 22:03:03.923868 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:03.923827    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 24 22:03:08.928631 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:08.928601    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:03:08.929130 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:08.929102    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 24 22:03:18.929159 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:18.929072    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 24 22:03:28.929572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:28.929527    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 24 22:03:38.929155 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:38.929113    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 24 22:03:48.930365 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:48.930336    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:03:56.257165 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:56.257131    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"]
Jul 24 22:03:56.257625 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:56.257492    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container" containerID="cri-o://59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9" gracePeriod=30
Jul 24 22:03:56.257625 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:56.257547    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kube-rbac-proxy" containerID="cri-o://32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73" gracePeriod=30
Jul 24 22:03:56.278337 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:56.278306    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:03:57.116536 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:57.116504    2585 generic.go:356] "Generic (PLEG): container finished" podID="55736fe1-104c-4dca-b699-89ea05ae2863" containerID="32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73" exitCode=2
Jul 24 22:03:57.116749 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:57.116578    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerDied","Data":"32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73"}
Jul 24 22:03:58.700523 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.700497    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:03:58.808642 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.808528    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-kube-rbac-proxy-sar-config\") pod \"55736fe1-104c-4dca-b699-89ea05ae2863\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") "
Jul 24 22:03:58.808642 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.808606    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55736fe1-104c-4dca-b699-89ea05ae2863-kserve-provision-location\") pod \"55736fe1-104c-4dca-b699-89ea05ae2863\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") "
Jul 24 22:03:58.808642 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.808637    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55736fe1-104c-4dca-b699-89ea05ae2863-proxy-tls\") pod \"55736fe1-104c-4dca-b699-89ea05ae2863\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") "
Jul 24 22:03:58.808943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.808694    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4gm7\" (UniqueName: \"kubernetes.io/projected/55736fe1-104c-4dca-b699-89ea05ae2863-kube-api-access-h4gm7\") pod \"55736fe1-104c-4dca-b699-89ea05ae2863\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") "
Jul 24 22:03:58.808943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.808718    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-cabundle-cert\") pod \"55736fe1-104c-4dca-b699-89ea05ae2863\" (UID: \"55736fe1-104c-4dca-b699-89ea05ae2863\") "
Jul 24 22:03:58.808943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.808925    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "55736fe1-104c-4dca-b699-89ea05ae2863" (UID: "55736fe1-104c-4dca-b699-89ea05ae2863"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:03:58.809152 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.809130    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55736fe1-104c-4dca-b699-89ea05ae2863" (UID: "55736fe1-104c-4dca-b699-89ea05ae2863"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:03:58.811023 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.810993    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55736fe1-104c-4dca-b699-89ea05ae2863-kube-api-access-h4gm7" (OuterVolumeSpecName: "kube-api-access-h4gm7") pod "55736fe1-104c-4dca-b699-89ea05ae2863" (UID: "55736fe1-104c-4dca-b699-89ea05ae2863"). InnerVolumeSpecName "kube-api-access-h4gm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:03:58.811146 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.811094    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55736fe1-104c-4dca-b699-89ea05ae2863-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55736fe1-104c-4dca-b699-89ea05ae2863" (UID: "55736fe1-104c-4dca-b699-89ea05ae2863"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:03:58.818248 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.818223    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55736fe1-104c-4dca-b699-89ea05ae2863-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55736fe1-104c-4dca-b699-89ea05ae2863" (UID: "55736fe1-104c-4dca-b699-89ea05ae2863"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:03:58.909654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.909612    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55736fe1-104c-4dca-b699-89ea05ae2863-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:03:58.909654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.909650    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55736fe1-104c-4dca-b699-89ea05ae2863-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:03:58.909654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.909679    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4gm7\" (UniqueName: \"kubernetes.io/projected/55736fe1-104c-4dca-b699-89ea05ae2863-kube-api-access-h4gm7\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:03:58.909874 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.909688    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:03:58.909874 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:58.909697    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55736fe1-104c-4dca-b699-89ea05ae2863-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:03:59.125075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.125036    2585 generic.go:356] "Generic (PLEG): container finished" podID="55736fe1-104c-4dca-b699-89ea05ae2863" containerID="59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9" exitCode=0
Jul 24 22:03:59.125075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.125075    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerDied","Data":"59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9"}
Jul 24 22:03:59.125267 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.125089    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"
Jul 24 22:03:59.125267 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.125121    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8" event={"ID":"55736fe1-104c-4dca-b699-89ea05ae2863","Type":"ContainerDied","Data":"1cff6bfc1a250226fff65b07456ca2c0f5aa3f553fa7903926813e995db046f7"}
Jul 24 22:03:59.125267 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.125148    2585 scope.go:117] "RemoveContainer" containerID="32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73"
Jul 24 22:03:59.134177 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.134160    2585 scope.go:117] "RemoveContainer" containerID="59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9"
Jul 24 22:03:59.141616 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.141600    2585 scope.go:117] "RemoveContainer" containerID="7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169"
Jul 24 22:03:59.147516 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.147493    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"]
Jul 24 22:03:59.149220 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.149203    2585 scope.go:117] "RemoveContainer" containerID="32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73"
Jul 24 22:03:59.149624 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:03:59.149531    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73\": container with ID starting with 32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73 not found: ID does not exist" containerID="32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73"
Jul 24 22:03:59.149624 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.149566    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73"} err="failed to get container status \"32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73\": rpc error: code = NotFound desc = could not find container \"32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73\": container with ID starting with 32425d6fb569661101cd551934d751cbc4ea5711084325f4a964c38161c69f73 not found: ID does not exist"
Jul 24 22:03:59.149624 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.149594    2585 scope.go:117] "RemoveContainer" containerID="59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9"
Jul 24 22:03:59.149966 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:03:59.149941    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9\": container with ID starting with 59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9 not found: ID does not exist" containerID="59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9"
Jul 24 22:03:59.150024 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.149978    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9"} err="failed to get container status \"59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9\": rpc error: code = NotFound desc = could not find container \"59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9\": container with ID starting with 59f7f1502d002ff65b0f6423eb3a618d1e328e54f0ad69262c87ba314273fbb9 not found: ID does not exist"
Jul 24 22:03:59.150024 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.150003    2585 scope.go:117] "RemoveContainer" containerID="7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169"
Jul 24 22:03:59.150351 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:03:59.150314    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169\": container with ID starting with 7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169 not found: ID does not exist" containerID="7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169"
Jul 24 22:03:59.150433 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.150358    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169"} err="failed to get container status \"7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169\": rpc error: code = NotFound desc = could not find container \"7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169\": container with ID starting with 7dd1510a9dbcb923dc0dd91755d57a1b9f6d9adaba5178d211d188c77790f169 not found: ID does not exist"
Jul 24 22:03:59.151427 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.151409    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-v9vk8"]
Jul 24 22:03:59.326858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:03:59.326828    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" path="/var/lib/kubelet/pods/55736fe1-104c-4dca-b699-89ea05ae2863/volumes"
Jul 24 22:04:01.392307 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:04:01.392261    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:01.563124 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:04:01.563094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:07.183418 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:04:07.183346    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:04:52.188201 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:04:52.188124    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:05:03.434292 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:03.434219    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:05:08.518388 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.518350    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:05:08.564016 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.563986    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"]
Jul 24 22:05:08.564391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564353    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kube-rbac-proxy"
Jul 24 22:05:08.564391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564379    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kube-rbac-proxy"
Jul 24 22:05:08.564391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564394    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="storage-initializer"
Jul 24 22:05:08.564647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564404    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="storage-initializer"
Jul 24 22:05:08.564647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564413    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container"
Jul 24 22:05:08.564647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564421    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container"
Jul 24 22:05:08.564647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564498    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kserve-container"
Jul 24 22:05:08.564647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.564509    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="55736fe1-104c-4dca-b699-89ea05ae2863" containerName="kube-rbac-proxy"
Jul 24 22:05:08.566920 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.566901    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.569593 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.569572    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 24 22:05:08.569741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.569723    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:05:08.569858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.569833    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:05:08.569858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.569853    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:05:08.570884 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.570860    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:05:08.570998 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.570906    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 24 22:05:08.570998 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.570947    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:05:08.578638 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.578615    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"]
Jul 24 22:05:08.600855 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.600825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4g9m\" (UniqueName: \"kubernetes.io/projected/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-api-access-s4g9m\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.600947 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.600861    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.600947 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.600880    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.601029 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.600957    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d504338-3d3a-405c-9823-79e2dea98a4d-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.601029 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.600973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701258    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d504338-3d3a-405c-9823-79e2dea98a4d-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701502 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4g9m\" (UniqueName: \"kubernetes.io/projected/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-api-access-s4g9m\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701502 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701398    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701502 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701700 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:05:08.701544    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 24 22:05:08.701700 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:05:08.701611    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls podName:1d504338-3d3a-405c-9823-79e2dea98a4d nodeName:}" failed. No retries permitted until 2026-07-24 22:05:09.201589327 +0000 UTC m=+2990.394767260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls") pod "isvc-pmml-predictor-8674795f59-cvmzv" (UID: "1d504338-3d3a-405c-9823-79e2dea98a4d") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 24 22:05:08.701848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701712    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d504338-3d3a-405c-9823-79e2dea98a4d-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.701988 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.701971    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.702048 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.702030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.710129 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.710107    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4g9m\" (UniqueName: \"kubernetes.io/projected/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-api-access-s4g9m\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:08.769544 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:08.769472    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:05:09.205373 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:09.205339    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:09.207925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:09.207906    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cvmzv\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:09.479820 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:09.479739    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:09.608971 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:09.608937    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"]
Jul 24 22:05:09.610827 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:05:09.610803    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d504338_3d3a_405c_9823_79e2dea98a4d.slice/crio-28000b97835671000a4bbcec1bb41c3745455ba63a5e17eca8d941ce694b4715 WatchSource:0}: Error finding container 28000b97835671000a4bbcec1bb41c3745455ba63a5e17eca8d941ce694b4715: Status 404 returned error can't find the container with id 28000b97835671000a4bbcec1bb41c3745455ba63a5e17eca8d941ce694b4715
Jul 24 22:05:10.376086 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:10.376046    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerStarted","Data":"88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001"}
Jul 24 22:05:10.376292 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:10.376155    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerStarted","Data":"28000b97835671000a4bbcec1bb41c3745455ba63a5e17eca8d941ce694b4715"}
Jul 24 22:05:14.390807 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:14.390768    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerID="88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001" exitCode=0
Jul 24 22:05:14.391213 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:14.390852    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerDied","Data":"88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001"}
Jul 24 22:05:14.392136 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:14.392119    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:05:21.424475 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:21.424450    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerStarted","Data":"95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525"}
Jul 24 22:05:22.430151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:22.430115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerStarted","Data":"d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55"}
Jul 24 22:05:22.430587 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:22.430275    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:22.430587 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:22.430314    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:22.431680 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:22.431643    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:05:22.452031 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:22.451985    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podStartSLOduration=3.764508796 podStartE2EDuration="14.451972351s" totalImagesPullingTime="6.916044012s" totalInitContainerRuntime="3.771419543s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:05:22.450424027 +0000 UTC m=+3003.643601983" watchObservedRunningTime="2026-07-24 22:05:22.451972351 +0000 UTC m=+3003.645150288"
Jul 24 22:05:23.433586 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:23.433539    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:05:28.437960 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:28.437927    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:05:28.442225 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:28.439304    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:05:38.438733 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:38.438693    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:05:48.438446 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:48.438407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:05:58.439054 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:05:58.439011    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:06:08.438452 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:08.438407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:06:18.438428 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:18.438349    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:06:28.439234 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:28.439196    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 24 22:06:36.322836 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:36.322802    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:06:39.648356 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:39.648324    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"]
Jul 24 22:06:39.648799 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:39.648539    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container" containerID="cri-o://95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525" gracePeriod=30
Jul 24 22:06:39.648799 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:39.648597    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kube-rbac-proxy" containerID="cri-o://d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55" gracePeriod=30
Jul 24 22:06:39.694961 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:39.694932    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:06:40.720526 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:40.720494    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerID="d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55" exitCode=2
Jul 24 22:06:40.720948 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:40.720586    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerDied","Data":"d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55"}
Jul 24 22:06:42.890557 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.890531    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:06:42.995531 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995451    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls\") pod \"1d504338-3d3a-405c-9823-79e2dea98a4d\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") "
Jul 24 22:06:42.995713 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995544    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-rbac-proxy-sar-config\") pod \"1d504338-3d3a-405c-9823-79e2dea98a4d\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") "
Jul 24 22:06:42.995713 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995570    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4g9m\" (UniqueName: \"kubernetes.io/projected/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-api-access-s4g9m\") pod \"1d504338-3d3a-405c-9823-79e2dea98a4d\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") "
Jul 24 22:06:42.995713 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995598    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d504338-3d3a-405c-9823-79e2dea98a4d-kserve-provision-location\") pod \"1d504338-3d3a-405c-9823-79e2dea98a4d\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") "
Jul 24 22:06:42.995713 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995630    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-cabundle-cert\") pod \"1d504338-3d3a-405c-9823-79e2dea98a4d\" (UID: \"1d504338-3d3a-405c-9823-79e2dea98a4d\") "
Jul 24 22:06:42.995945 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995920    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d504338-3d3a-405c-9823-79e2dea98a4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d504338-3d3a-405c-9823-79e2dea98a4d" (UID: "1d504338-3d3a-405c-9823-79e2dea98a4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:06:42.996002 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.995968    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d504338-3d3a-405c-9823-79e2dea98a4d" (UID: "1d504338-3d3a-405c-9823-79e2dea98a4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:06:42.996066 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.996005    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d504338-3d3a-405c-9823-79e2dea98a4d" (UID: "1d504338-3d3a-405c-9823-79e2dea98a4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:06:42.997772 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.997748    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-api-access-s4g9m" (OuterVolumeSpecName: "kube-api-access-s4g9m") pod "1d504338-3d3a-405c-9823-79e2dea98a4d" (UID: "1d504338-3d3a-405c-9823-79e2dea98a4d"). InnerVolumeSpecName "kube-api-access-s4g9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:06:42.997834 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:42.997802    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d504338-3d3a-405c-9823-79e2dea98a4d" (UID: "1d504338-3d3a-405c-9823-79e2dea98a4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:06:43.096858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.096836    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d504338-3d3a-405c-9823-79e2dea98a4d-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:06:43.096858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.096856    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:06:43.097007 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.096866    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4g9m\" (UniqueName: \"kubernetes.io/projected/1d504338-3d3a-405c-9823-79e2dea98a4d-kube-api-access-s4g9m\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:06:43.097007 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.096874    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d504338-3d3a-405c-9823-79e2dea98a4d-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:06:43.097007 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.096883    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d504338-3d3a-405c-9823-79e2dea98a4d-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:06:43.732859 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.732822    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerID="95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525" exitCode=0
Jul 24 22:06:43.733056 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.732872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerDied","Data":"95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525"}
Jul 24 22:06:43.733056 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.732905    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"
Jul 24 22:06:43.733056 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.732922    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv" event={"ID":"1d504338-3d3a-405c-9823-79e2dea98a4d","Type":"ContainerDied","Data":"28000b97835671000a4bbcec1bb41c3745455ba63a5e17eca8d941ce694b4715"}
Jul 24 22:06:43.733056 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.732949    2585 scope.go:117] "RemoveContainer" containerID="d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55"
Jul 24 22:06:43.742700 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.742654    2585 scope.go:117] "RemoveContainer" containerID="95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525"
Jul 24 22:06:43.749778 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.749758    2585 scope.go:117] "RemoveContainer" containerID="88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001"
Jul 24 22:06:43.753377 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.753353    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"]
Jul 24 22:06:43.756629 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.756609    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cvmzv"]
Jul 24 22:06:43.758266 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.758239    2585 scope.go:117] "RemoveContainer" containerID="d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55"
Jul 24 22:06:43.758552 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:06:43.758531    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55\": container with ID starting with d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55 not found: ID does not exist" containerID="d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55"
Jul 24 22:06:43.758616 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.758560    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55"} err="failed to get container status \"d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55\": rpc error: code = NotFound desc = could not find container \"d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55\": container with ID starting with d3071ee00db0e7c2a2b4a89015720bff74d203bed1bf125dabd1b17b0f94bd55 not found: ID does not exist"
Jul 24 22:06:43.758616 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.758579    2585 scope.go:117] "RemoveContainer" containerID="95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525"
Jul 24 22:06:43.758893 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:06:43.758863    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525\": container with ID starting with 95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525 not found: ID does not exist" containerID="95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525"
Jul 24 22:06:43.758958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.758891    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525"} err="failed to get container status \"95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525\": rpc error: code = NotFound desc = could not find container \"95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525\": container with ID starting with 95efb6d87b856aa17db86ee187fe1ad42ca9f98b2de48a492286214b817c2525 not found: ID does not exist"
Jul 24 22:06:43.758958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.758909    2585 scope.go:117] "RemoveContainer" containerID="88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001"
Jul 24 22:06:43.759152 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:06:43.759135    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001\": container with ID starting with 88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001 not found: ID does not exist" containerID="88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001"
Jul 24 22:06:43.759193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:43.759160    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001"} err="failed to get container status \"88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001\": rpc error: code = NotFound desc = could not find container \"88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001\": container with ID starting with 88533dc630a969b9290ebd98b30c778aebf4a6e3e566bd56884f470dfea5f001 not found: ID does not exist"
Jul 24 22:06:44.666808 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.666779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:06:44.739606 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739559    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"]
Jul 24 22:06:44.739886 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739868    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kube-rbac-proxy"
Jul 24 22:06:44.739886 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739885    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kube-rbac-proxy"
Jul 24 22:06:44.740060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739900    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container"
Jul 24 22:06:44.740060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739910    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container"
Jul 24 22:06:44.740060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739927    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="storage-initializer"
Jul 24 22:06:44.740060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.739935    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="storage-initializer"
Jul 24 22:06:44.740060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.740019    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kube-rbac-proxy"
Jul 24 22:06:44.740060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.740035    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" containerName="kserve-container"
Jul 24 22:06:44.744358 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.744336    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.747083 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.747062    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 24 22:06:44.747202 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.747067    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:06:44.747389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.747368    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:06:44.747389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.747381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:06:44.747551 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.747395    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:06:44.747551 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.747395    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:06:44.748251 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.748235    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:06:44.753435 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.753398    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"]
Jul 24 22:06:44.809036 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.809008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.809152 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.809049    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.809205 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.809140    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-api-access-d7xs6\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.809205 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.809178    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6793e030-9b53-4446-ab66-bc1f3a39ba71-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.809292 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.809207    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6793e030-9b53-4446-ab66-bc1f3a39ba71-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.909792 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.909759    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.909926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.909797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.909926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.909860    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-api-access-d7xs6\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.909926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.909892    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6793e030-9b53-4446-ab66-bc1f3a39ba71-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.909926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.909922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6793e030-9b53-4446-ab66-bc1f3a39ba71-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.910312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.910292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6793e030-9b53-4446-ab66-bc1f3a39ba71-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.910569 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.910548    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.910698 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.910585    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.912428 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.912408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6793e030-9b53-4446-ab66-bc1f3a39ba71-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:44.916204 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.916179    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:06:44.919171 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:44.919104    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-api-access-d7xs6\") pod \"isvc-pmml-runtime-predictor-885fc8777-nmh2s\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:45.057425 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:45.057399    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:45.186877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:45.186844    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"]
Jul 24 22:06:45.189299 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:06:45.189273    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6793e030_9b53_4446_ab66_bc1f3a39ba71.slice/crio-aa924ca1e0594f36068fbed28ba79713b4c9b5e374a15ad9a93433ca43e94001 WatchSource:0}: Error finding container aa924ca1e0594f36068fbed28ba79713b4c9b5e374a15ad9a93433ca43e94001: Status 404 returned error can't find the container with id aa924ca1e0594f36068fbed28ba79713b4c9b5e374a15ad9a93433ca43e94001
Jul 24 22:06:45.327497 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:45.327463    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d504338-3d3a-405c-9823-79e2dea98a4d" path="/var/lib/kubelet/pods/1d504338-3d3a-405c-9823-79e2dea98a4d/volumes"
Jul 24 22:06:45.744890 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:45.744852    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerStarted","Data":"c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435"}
Jul 24 22:06:45.745270 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:45.744925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerStarted","Data":"aa924ca1e0594f36068fbed28ba79713b4c9b5e374a15ad9a93433ca43e94001"}
Jul 24 22:06:49.760957 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:49.760925    2585 generic.go:356] "Generic (PLEG): container finished" podID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerID="c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435" exitCode=0
Jul 24 22:06:49.761361 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:49.760975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerDied","Data":"c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435"}
Jul 24 22:06:50.770680 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:50.770626    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerStarted","Data":"6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee"}
Jul 24 22:06:50.771061 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:50.770708    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerStarted","Data":"947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae"}
Jul 24 22:06:50.771061 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:50.770975    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:50.771147 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:50.771113    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:50.772376 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:50.772353    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:06:50.794076 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:50.794022    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podStartSLOduration=3.220328988 podStartE2EDuration="6.79400934s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.573680352s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:06:50.791546036 +0000 UTC m=+3091.984723997" watchObservedRunningTime="2026-07-24 22:06:50.79400934 +0000 UTC m=+3091.987187276"
Jul 24 22:06:51.773998 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:51.773960    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:06:56.778702 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:56.778651    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:06:56.779151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:06:56.779124    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:07:06.780091 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:07:06.780047    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:07:16.779240 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:07:16.779198    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:07:26.779926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:07:26.779881    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:07:36.780126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:07:36.780072    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:07:46.780027 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:07:46.779971    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:07:56.779432 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:07:56.779394    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 24 22:08:04.322883 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:04.322849    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:08:05.745884 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:05.745848    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"]
Jul 24 22:08:05.769421 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:05.769381    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:08:06.043724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:06.043436    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container" containerID="cri-o://947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae" gracePeriod=30
Jul 24 22:08:06.043724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:06.043502    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kube-rbac-proxy" containerID="cri-o://6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee" gracePeriod=30
Jul 24 22:08:06.774454 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:06.774401    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 24 22:08:07.049040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:07.048955    2585 generic.go:356] "Generic (PLEG): container finished" podID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerID="6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee" exitCode=2
Jul 24 22:08:07.049040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:07.049028    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerDied","Data":"6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee"}
Jul 24 22:08:09.587132 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.587108    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:08:09.636860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.636793    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-rbac-proxy-sar-config\") pod \"6793e030-9b53-4446-ab66-bc1f3a39ba71\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") "
Jul 24 22:08:09.636860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.636847    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-api-access-d7xs6\") pod \"6793e030-9b53-4446-ab66-bc1f3a39ba71\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") "
Jul 24 22:08:09.637054 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.636868    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6793e030-9b53-4446-ab66-bc1f3a39ba71-proxy-tls\") pod \"6793e030-9b53-4446-ab66-bc1f3a39ba71\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") "
Jul 24 22:08:09.637054 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.636890    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-cabundle-cert\") pod \"6793e030-9b53-4446-ab66-bc1f3a39ba71\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") "
Jul 24 22:08:09.637054 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.637018    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6793e030-9b53-4446-ab66-bc1f3a39ba71-kserve-provision-location\") pod \"6793e030-9b53-4446-ab66-bc1f3a39ba71\" (UID: \"6793e030-9b53-4446-ab66-bc1f3a39ba71\") "
Jul 24 22:08:09.637294 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.637213    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6793e030-9b53-4446-ab66-bc1f3a39ba71" (UID: "6793e030-9b53-4446-ab66-bc1f3a39ba71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:08:09.637383 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.637312    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6793e030-9b53-4446-ab66-bc1f3a39ba71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6793e030-9b53-4446-ab66-bc1f3a39ba71" (UID: "6793e030-9b53-4446-ab66-bc1f3a39ba71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:08:09.637383 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.637203    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6793e030-9b53-4446-ab66-bc1f3a39ba71" (UID: "6793e030-9b53-4446-ab66-bc1f3a39ba71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:08:09.639040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.639022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6793e030-9b53-4446-ab66-bc1f3a39ba71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6793e030-9b53-4446-ab66-bc1f3a39ba71" (UID: "6793e030-9b53-4446-ab66-bc1f3a39ba71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:08:09.639126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.639085    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-api-access-d7xs6" (OuterVolumeSpecName: "kube-api-access-d7xs6") pod "6793e030-9b53-4446-ab66-bc1f3a39ba71" (UID: "6793e030-9b53-4446-ab66-bc1f3a39ba71"). InnerVolumeSpecName "kube-api-access-d7xs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:08:09.738480 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.738458    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6793e030-9b53-4446-ab66-bc1f3a39ba71-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:08:09.738480 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.738478    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:08:09.738617 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.738487    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7xs6\" (UniqueName: \"kubernetes.io/projected/6793e030-9b53-4446-ab66-bc1f3a39ba71-kube-api-access-d7xs6\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:08:09.738617 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.738497    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6793e030-9b53-4446-ab66-bc1f3a39ba71-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:08:09.738617 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:09.738506    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6793e030-9b53-4446-ab66-bc1f3a39ba71-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:08:10.059842 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.059756    2585 generic.go:356] "Generic (PLEG): container finished" podID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerID="947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae" exitCode=0
Jul 24 22:08:10.059842 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.059813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerDied","Data":"947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae"}
Jul 24 22:08:10.059842 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.059836    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"
Jul 24 22:08:10.060044 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.059905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s" event={"ID":"6793e030-9b53-4446-ab66-bc1f3a39ba71","Type":"ContainerDied","Data":"aa924ca1e0594f36068fbed28ba79713b4c9b5e374a15ad9a93433ca43e94001"}
Jul 24 22:08:10.060044 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.059946    2585 scope.go:117] "RemoveContainer" containerID="6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee"
Jul 24 22:08:10.068522 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.068502    2585 scope.go:117] "RemoveContainer" containerID="947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae"
Jul 24 22:08:10.077196 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.077177    2585 scope.go:117] "RemoveContainer" containerID="c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435"
Jul 24 22:08:10.082482 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.082457    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"]
Jul 24 22:08:10.085274 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.085256    2585 scope.go:117] "RemoveContainer" containerID="6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee"
Jul 24 22:08:10.085335 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.085315    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-nmh2s"]
Jul 24 22:08:10.085518 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:08:10.085500    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee\": container with ID starting with 6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee not found: ID does not exist" containerID="6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee"
Jul 24 22:08:10.085560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.085540    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee"} err="failed to get container status \"6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee\": rpc error: code = NotFound desc = could not find container \"6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee\": container with ID starting with 6c6b58f36a10dfbdba015ab241c282c8a8df38911b3569438152fc7e7adcd5ee not found: ID does not exist"
Jul 24 22:08:10.085560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.085557    2585 scope.go:117] "RemoveContainer" containerID="947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae"
Jul 24 22:08:10.085819 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:08:10.085802    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae\": container with ID starting with 947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae not found: ID does not exist" containerID="947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae"
Jul 24 22:08:10.085881 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.085825    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae"} err="failed to get container status \"947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae\": rpc error: code = NotFound desc = could not find container \"947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae\": container with ID starting with 947ca21567076dcf6ac3765d6ee15126481da283cffd6f94f829b3e8525abcae not found: ID does not exist"
Jul 24 22:08:10.085881 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.085840    2585 scope.go:117] "RemoveContainer" containerID="c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435"
Jul 24 22:08:10.086058 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:08:10.086042    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435\": container with ID starting with c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435 not found: ID does not exist" containerID="c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435"
Jul 24 22:08:10.086097 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.086061    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435"} err="failed to get container status \"c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435\": rpc error: code = NotFound desc = could not find container \"c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435\": container with ID starting with c6e229db280e3050628fc6d051dce77c37db7a838921354af96fd45a69407435 not found: ID does not exist"
Jul 24 22:08:10.870012 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.869978    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:08:10.917878 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.917844    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"]
Jul 24 22:08:10.918138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918124    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="storage-initializer"
Jul 24 22:08:10.918181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918143    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="storage-initializer"
Jul 24 22:08:10.918181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918159    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container"
Jul 24 22:08:10.918181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918164    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container"
Jul 24 22:08:10.918181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918175    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kube-rbac-proxy"
Jul 24 22:08:10.918181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918181    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kube-rbac-proxy"
Jul 24 22:08:10.918346 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918231    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kserve-container"
Jul 24 22:08:10.918346 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.918241    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" containerName="kube-rbac-proxy"
Jul 24 22:08:10.922750 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.922731    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:10.925593 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.925571    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 24 22:08:10.925838 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.925822    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:08:10.925926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.925892    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:08:10.925926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.925917    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:08:10.926868 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.926850    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:08:10.926967 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.926941    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:08:10.926967 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.926950    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 24 22:08:10.934183 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.934164    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"]
Jul 24 22:08:10.946643 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.946623    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:10.946817 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.946756    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:10.946880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.946820    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:10.946880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.946853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:10.946990 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:10.946886    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvd7\" (UniqueName: \"kubernetes.io/projected/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-api-access-kkvd7\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.022260 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.022229    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:08:11.047384 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.047352    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.047497 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.047399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.047497 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.047429    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.047497 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.047455    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvd7\" (UniqueName: \"kubernetes.io/projected/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-api-access-kkvd7\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.047497 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.047491    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.047927 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.047906    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.048167 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.048151    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.048218 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.048178    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.049954 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.049937    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.055480 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.055461    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvd7\" (UniqueName: \"kubernetes.io/projected/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-api-access-kkvd7\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.233921 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.233844    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:11.327060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.327031    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6793e030-9b53-4446-ab66-bc1f3a39ba71" path="/var/lib/kubelet/pods/6793e030-9b53-4446-ab66-bc1f3a39ba71/volumes"
Jul 24 22:08:11.362425 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:11.362397    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"]
Jul 24 22:08:11.365273 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:08:11.365245    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2733bd53_fdf0_4c84_b46e_65ba74fc7d56.slice/crio-d85fce0b8a6fc980bcd46235536550f7c43c3fb0e174f3e465d29b2313d65dbf WatchSource:0}: Error finding container d85fce0b8a6fc980bcd46235536550f7c43c3fb0e174f3e465d29b2313d65dbf: Status 404 returned error can't find the container with id d85fce0b8a6fc980bcd46235536550f7c43c3fb0e174f3e465d29b2313d65dbf
Jul 24 22:08:12.070361 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:12.070327    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerStarted","Data":"6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399"}
Jul 24 22:08:12.070710 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:12.070392    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerStarted","Data":"d85fce0b8a6fc980bcd46235536550f7c43c3fb0e174f3e465d29b2313d65dbf"}
Jul 24 22:08:16.089222 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:16.089184    2585 generic.go:356] "Generic (PLEG): container finished" podID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerID="6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399" exitCode=0
Jul 24 22:08:16.089593 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:16.089242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerDied","Data":"6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399"}
Jul 24 22:08:17.098403 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:17.098364    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerStarted","Data":"5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c"}
Jul 24 22:08:17.098874 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:17.098428    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerStarted","Data":"23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da"}
Jul 24 22:08:17.098874 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:17.098714    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:17.098983 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:17.098885    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:17.100065 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:17.100044    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:08:17.120983 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:17.120936    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podStartSLOduration=3.23892791 podStartE2EDuration="7.12092234s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88199443s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:08:17.119988327 +0000 UTC m=+3178.313166265" watchObservedRunningTime="2026-07-24 22:08:17.12092234 +0000 UTC m=+3178.314100276"
Jul 24 22:08:18.102535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:18.102497    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:08:19.106284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:19.106248    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:08:24.110563 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:24.110531    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:08:24.111217 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:24.111185    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:08:34.111496 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:34.111456    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:08:44.111056 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:44.111010    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:08:54.111912 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:08:54.111863    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:09:04.111358 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:04.111319    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:09:14.111969 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:14.111884    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:09:24.111237 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:24.111186    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:09:34.112311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:34.112280    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:09:42.185505 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:42.185465    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"]
Jul 24 22:09:42.186004 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:42.185729    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" containerID="cri-o://23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da" gracePeriod=30
Jul 24 22:09:42.186004 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:42.185783    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kube-rbac-proxy" containerID="cri-o://5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c" gracePeriod=30
Jul 24 22:09:42.242334 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:42.242301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:09:42.396520 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:42.396491    2585 generic.go:356] "Generic (PLEG): container finished" podID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerID="5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c" exitCode=2
Jul 24 22:09:42.396694 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:42.396535    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerDied","Data":"5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c"}
Jul 24 22:09:44.107024 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:44.106982    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 24 22:09:44.111438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:44.111407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 24 22:09:45.629945 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.629923    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:09:45.733060 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733033    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-rbac-proxy-sar-config\") pod \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") "
Jul 24 22:09:45.733247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733089    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kserve-provision-location\") pod \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") "
Jul 24 22:09:45.733247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733120    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-proxy-tls\") pod \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") "
Jul 24 22:09:45.733247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733139    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-cabundle-cert\") pod \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") "
Jul 24 22:09:45.733247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733163    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkvd7\" (UniqueName: \"kubernetes.io/projected/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-api-access-kkvd7\") pod \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\" (UID: \"2733bd53-fdf0-4c84-b46e-65ba74fc7d56\") "
Jul 24 22:09:45.733534 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733509    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2733bd53-fdf0-4c84-b46e-65ba74fc7d56" (UID: "2733bd53-fdf0-4c84-b46e-65ba74fc7d56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:09:45.733601 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733543    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2733bd53-fdf0-4c84-b46e-65ba74fc7d56" (UID: "2733bd53-fdf0-4c84-b46e-65ba74fc7d56"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:09:45.733685 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.733648    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2733bd53-fdf0-4c84-b46e-65ba74fc7d56" (UID: "2733bd53-fdf0-4c84-b46e-65ba74fc7d56"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:09:45.735465 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.735444    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-api-access-kkvd7" (OuterVolumeSpecName: "kube-api-access-kkvd7") pod "2733bd53-fdf0-4c84-b46e-65ba74fc7d56" (UID: "2733bd53-fdf0-4c84-b46e-65ba74fc7d56"). InnerVolumeSpecName "kube-api-access-kkvd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:09:45.735465 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.735457    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2733bd53-fdf0-4c84-b46e-65ba74fc7d56" (UID: "2733bd53-fdf0-4c84-b46e-65ba74fc7d56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:09:45.833633 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.833586    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvd7\" (UniqueName: \"kubernetes.io/projected/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-api-access-kkvd7\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:09:45.833633 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.833628    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:09:45.833633 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.833639    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:09:45.833633 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.833648    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:09:45.833633 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:45.833657    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2733bd53-fdf0-4c84-b46e-65ba74fc7d56-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:09:46.412638 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.412603    2585 generic.go:356] "Generic (PLEG): container finished" podID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerID="23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da" exitCode=0
Jul 24 22:09:46.412819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.412654    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"
Jul 24 22:09:46.412819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.412690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerDied","Data":"23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da"}
Jul 24 22:09:46.412819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.412752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt" event={"ID":"2733bd53-fdf0-4c84-b46e-65ba74fc7d56","Type":"ContainerDied","Data":"d85fce0b8a6fc980bcd46235536550f7c43c3fb0e174f3e465d29b2313d65dbf"}
Jul 24 22:09:46.412819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.412779    2585 scope.go:117] "RemoveContainer" containerID="5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c"
Jul 24 22:09:46.420944 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.420928    2585 scope.go:117] "RemoveContainer" containerID="23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da"
Jul 24 22:09:46.427951 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.427935    2585 scope.go:117] "RemoveContainer" containerID="6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399"
Jul 24 22:09:46.433928 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.433907    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"]
Jul 24 22:09:46.435220 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.435201    2585 scope.go:117] "RemoveContainer" containerID="5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c"
Jul 24 22:09:46.435519 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:09:46.435499    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c\": container with ID starting with 5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c not found: ID does not exist" containerID="5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c"
Jul 24 22:09:46.435634 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.435558    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c"} err="failed to get container status \"5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c\": rpc error: code = NotFound desc = could not find container \"5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c\": container with ID starting with 5be60fdaaa892b8a5efcf90f11e5757b24ce5c1d8a67e1594c2f8f13ba209a1c not found: ID does not exist"
Jul 24 22:09:46.435634 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.435591    2585 scope.go:117] "RemoveContainer" containerID="23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da"
Jul 24 22:09:46.436041 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:09:46.436012    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da\": container with ID starting with 23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da not found: ID does not exist" containerID="23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da"
Jul 24 22:09:46.436149 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.436087    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da"} err="failed to get container status \"23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da\": rpc error: code = NotFound desc = could not find container \"23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da\": container with ID starting with 23c2a774da9c98742c82944133f66b3292202e400835c2b52d762daa6ee068da not found: ID does not exist"
Jul 24 22:09:46.436149 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.436115    2585 scope.go:117] "RemoveContainer" containerID="6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399"
Jul 24 22:09:46.436603 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:09:46.436562    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399\": container with ID starting with 6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399 not found: ID does not exist" containerID="6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399"
Jul 24 22:09:46.436700 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.436612    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399"} err="failed to get container status \"6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399\": rpc error: code = NotFound desc = could not find container \"6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399\": container with ID starting with 6aebc136212c2c5b95782fa4542bde03cb9d9508ae629af42b0afb2dd9f28399 not found: ID does not exist"
Jul 24 22:09:46.437570 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:46.437550    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-2z5zt"]
Jul 24 22:09:47.327762 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.327730    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" path="/var/lib/kubelet/pods/2733bd53-fdf0-4c84-b46e-65ba74fc7d56/volumes"
Jul 24 22:09:47.347494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.347461    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:09:47.437318 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437284    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"]
Jul 24 22:09:47.437559 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437547    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container"
Jul 24 22:09:47.437623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437560    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container"
Jul 24 22:09:47.437623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437570    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kube-rbac-proxy"
Jul 24 22:09:47.437623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437575    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kube-rbac-proxy"
Jul 24 22:09:47.437623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437586    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="storage-initializer"
Jul 24 22:09:47.437623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437591    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="storage-initializer"
Jul 24 22:09:47.437909 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437639    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kube-rbac-proxy"
Jul 24 22:09:47.437909 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.437648    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2733bd53-fdf0-4c84-b46e-65ba74fc7d56" containerName="kserve-container"
Jul 24 22:09:47.442267 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.442247    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.445503 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.445475    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:09:47.445648 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.445585    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:09:47.445749 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.445655    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:09:47.445749 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.445686    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:09:47.446794 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.446765    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5c6eb3-predictor-serving-cert\""
Jul 24 22:09:47.446794 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.446788    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:09:47.446794 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.446772    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-5c6eb3-kube-rbac-proxy-sar-config\""
Jul 24 22:09:47.456834 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.456811    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"]
Jul 24 22:09:47.544267 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.544238    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kserve-provision-location\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.544442 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.544279    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.544442 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.544303    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.544442 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.544326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-cabundle-cert\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.544442 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.544368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5nhh\" (UniqueName: \"kubernetes.io/projected/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-api-access-s5nhh\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.645655 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.645625    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kserve-provision-location\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.645841 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.645698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.645841 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.645725    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.645841 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.645751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-cabundle-cert\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.646022 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:09:47.645846    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-serving-cert: secret "isvc-primary-5c6eb3-predictor-serving-cert" not found
Jul 24 22:09:47.646022 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.645867    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nhh\" (UniqueName: \"kubernetes.io/projected/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-api-access-s5nhh\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.646022 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:09:47.645909    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls podName:d273ef49-6601-4b1b-a1fc-19e9bd444c7e nodeName:}" failed. No retries permitted until 2026-07-24 22:09:48.145887875 +0000 UTC m=+3269.339065792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls") pod "isvc-primary-5c6eb3-predictor-5587458446-fllrp" (UID: "d273ef49-6601-4b1b-a1fc-19e9bd444c7e") : secret "isvc-primary-5c6eb3-predictor-serving-cert" not found
Jul 24 22:09:47.646022 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.646002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kserve-provision-location\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.646463 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.646441    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.646534 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.646481    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-cabundle-cert\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.655931 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.655912    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nhh\" (UniqueName: \"kubernetes.io/projected/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-api-access-s5nhh\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:47.721768 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:47.721729    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:09:48.148043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:48.147997    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:48.150678 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:48.150627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls\") pod \"isvc-primary-5c6eb3-predictor-5587458446-fllrp\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") " pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:48.352896 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:48.352848    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:48.482209 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:48.482186    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"]
Jul 24 22:09:48.484520 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:09:48.484493    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd273ef49_6601_4b1b_a1fc_19e9bd444c7e.slice/crio-893abbb9da6e793611fdeaa3292bd18344843e94125ac8d8f86127d1e1c943e6 WatchSource:0}: Error finding container 893abbb9da6e793611fdeaa3292bd18344843e94125ac8d8f86127d1e1c943e6: Status 404 returned error can't find the container with id 893abbb9da6e793611fdeaa3292bd18344843e94125ac8d8f86127d1e1c943e6
Jul 24 22:09:49.431724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:49.431690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerStarted","Data":"bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73"}
Jul 24 22:09:49.432131 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:49.431749    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerStarted","Data":"893abbb9da6e793611fdeaa3292bd18344843e94125ac8d8f86127d1e1c943e6"}
Jul 24 22:09:52.442747 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:52.442718    2585 generic.go:356] "Generic (PLEG): container finished" podID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerID="bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73" exitCode=0
Jul 24 22:09:52.443099 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:52.442758    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerDied","Data":"bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73"}
Jul 24 22:09:53.448661 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:53.448625    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerStarted","Data":"28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47"}
Jul 24 22:09:53.449049 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:53.448706    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerStarted","Data":"73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f"}
Jul 24 22:09:53.449049 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:53.448898    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:53.468888 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:53.468843    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podStartSLOduration=2.638318982 podStartE2EDuration="6.468830919s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830511937s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:09:53.467480644 +0000 UTC m=+3274.660658600" watchObservedRunningTime="2026-07-24 22:09:53.468830919 +0000 UTC m=+3274.662008854"
Jul 24 22:09:54.452237 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:54.452202    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:09:54.453495 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:54.453469    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:09:55.456393 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:09:55.456350    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:10:00.460657 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:00.460628    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:10:00.461227 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:00.461197    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:10:02.017449 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:02.017411    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:10:10.461327 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:10.461275    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:10:20.461717 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:20.461648    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:10:30.461994 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:30.461955    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:10:40.461452 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:40.461362    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:10:50.462063 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:10:50.462017    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:11:00.461356 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:00.461316    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 24 22:11:10.461837 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:10.461799    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:11:32.594288 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.594253    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"]
Jul 24 22:11:32.597308 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.597284    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.599977 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.599954    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-5c6eb3\""
Jul 24 22:11:32.600112 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.600070    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5c6eb3-kube-rbac-proxy-sar-config\""
Jul 24 22:11:32.600219 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.600108    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-5c6eb3-dockercfg-tnwqk\""
Jul 24 22:11:32.600396 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.600377    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5c6eb3-predictor-serving-cert\""
Jul 24 22:11:32.610650 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.610629    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"]
Jul 24 22:11:32.710050 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.710018    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92n8\" (UniqueName: \"kubernetes.io/projected/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-api-access-k92n8\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.710231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.710059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.710231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.710088    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-cabundle-cert\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.710231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.710204    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3d0eaa-76c8-478d-ba9d-efd09407cde7-proxy-tls\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.710344 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.710234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kserve-provision-location\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.810709 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.810654    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92n8\" (UniqueName: \"kubernetes.io/projected/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-api-access-k92n8\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.810889 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.810717    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.810889 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.810736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-cabundle-cert\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.810889 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.810788    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3d0eaa-76c8-478d-ba9d-efd09407cde7-proxy-tls\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.810889 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.810807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kserve-provision-location\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.811230 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.811208    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kserve-provision-location\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.811407 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.811382    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.811515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.811487    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-cabundle-cert\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.813459 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.813443    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3d0eaa-76c8-478d-ba9d-efd09407cde7-proxy-tls\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.820441 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.820421    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92n8\" (UniqueName: \"kubernetes.io/projected/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-api-access-k92n8\") pod \"isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") " pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:32.906975 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:32.906938    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:33.045386 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:33.045358    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"]
Jul 24 22:11:33.047946 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:11:33.047907    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3d0eaa_76c8_478d_ba9d_efd09407cde7.slice/crio-e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e WatchSource:0}: Error finding container e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e: Status 404 returned error can't find the container with id e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e
Jul 24 22:11:33.436165 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:33.436127    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:11:33.488864 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:33.488830    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:11:33.812597 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:33.812497    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" event={"ID":"de3d0eaa-76c8-478d-ba9d-efd09407cde7","Type":"ContainerStarted","Data":"6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e"}
Jul 24 22:11:33.812597 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:33.812562    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" event={"ID":"de3d0eaa-76c8-478d-ba9d-efd09407cde7","Type":"ContainerStarted","Data":"e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e"}
Jul 24 22:11:39.835746 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:39.835692    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/0.log"
Jul 24 22:11:39.836137 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:39.835757    2585 generic.go:356] "Generic (PLEG): container finished" podID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerID="6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e" exitCode=1
Jul 24 22:11:39.836137 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:39.835794    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" event={"ID":"de3d0eaa-76c8-478d-ba9d-efd09407cde7","Type":"ContainerDied","Data":"6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e"}
Jul 24 22:11:40.841932 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:40.841904    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/0.log"
Jul 24 22:11:40.842314 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:40.842018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" event={"ID":"de3d0eaa-76c8-478d-ba9d-efd09407cde7","Type":"ContainerStarted","Data":"2bfb65cf2d1b28a903a19a8145fd30ae6ecf7eeddcaf6555db664e042d881e58"}
Jul 24 22:11:45.861294 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.861260    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/1.log"
Jul 24 22:11:45.861726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.861709    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/0.log"
Jul 24 22:11:45.861814 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.861751    2585 generic.go:356] "Generic (PLEG): container finished" podID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerID="2bfb65cf2d1b28a903a19a8145fd30ae6ecf7eeddcaf6555db664e042d881e58" exitCode=1
Jul 24 22:11:45.861814 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.861797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" event={"ID":"de3d0eaa-76c8-478d-ba9d-efd09407cde7","Type":"ContainerDied","Data":"2bfb65cf2d1b28a903a19a8145fd30ae6ecf7eeddcaf6555db664e042d881e58"}
Jul 24 22:11:45.861919 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.861867    2585 scope.go:117] "RemoveContainer" containerID="6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e"
Jul 24 22:11:45.862197 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.862175    2585 scope.go:117] "RemoveContainer" containerID="6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e"
Jul 24 22:11:45.872537 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:11:45.872513    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_kserve-ci-e2e-test(de3d0eaa-76c8-478d-ba9d-efd09407cde7)\"" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7"
Jul 24 22:11:45.872884 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:11:45.872860    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_kserve-ci-e2e-test_de3d0eaa-76c8-478d-ba9d-efd09407cde7_0 in pod sandbox e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e from index: no such id: '6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e'" containerID="6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e"
Jul 24 22:11:45.872979 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:45.872890    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_kserve-ci-e2e-test_de3d0eaa-76c8-478d-ba9d-efd09407cde7_0 in pod sandbox e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e from index: no such id: '6b8005cca9f7e3650912c6ce93097bd9927b3ad01360324342eae28c4154338e'"
Jul 24 22:11:46.867132 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:46.867100    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/1.log"
Jul 24 22:11:50.597054 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.597021    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"]
Jul 24 22:11:50.597527 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.597335    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container" containerID="cri-o://73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f" gracePeriod=30
Jul 24 22:11:50.597527 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.597364    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kube-rbac-proxy" containerID="cri-o://28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47" gracePeriod=30
Jul 24 22:11:50.622041 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.622004    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:11:50.649008 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.648678    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"]
Jul 24 22:11:50.672862 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.672827    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:11:50.775754 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.775734    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/1.log"
Jul 24 22:11:50.777914 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.777898    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:50.846397 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846370    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kserve-provision-location\") pod \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") "
Jul 24 22:11:50.846560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846411    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3d0eaa-76c8-478d-ba9d-efd09407cde7-proxy-tls\") pod \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") "
Jul 24 22:11:50.846560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846450    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-cabundle-cert\") pod \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") "
Jul 24 22:11:50.846560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846473    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-rbac-proxy-sar-config\") pod \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") "
Jul 24 22:11:50.846560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846533    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k92n8\" (UniqueName: \"kubernetes.io/projected/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-api-access-k92n8\") pod \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\" (UID: \"de3d0eaa-76c8-478d-ba9d-efd09407cde7\") "
Jul 24 22:11:50.846838 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846651    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de3d0eaa-76c8-478d-ba9d-efd09407cde7" (UID: "de3d0eaa-76c8-478d-ba9d-efd09407cde7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:11:50.846838 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846817    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "de3d0eaa-76c8-478d-ba9d-efd09407cde7" (UID: "de3d0eaa-76c8-478d-ba9d-efd09407cde7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:11:50.846920 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846824    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:50.846920 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.846844    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "de3d0eaa-76c8-478d-ba9d-efd09407cde7" (UID: "de3d0eaa-76c8-478d-ba9d-efd09407cde7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:11:50.848802 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.848741    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-api-access-k92n8" (OuterVolumeSpecName: "kube-api-access-k92n8") pod "de3d0eaa-76c8-478d-ba9d-efd09407cde7" (UID: "de3d0eaa-76c8-478d-ba9d-efd09407cde7"). InnerVolumeSpecName "kube-api-access-k92n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:11:50.848887 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.848865    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3d0eaa-76c8-478d-ba9d-efd09407cde7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de3d0eaa-76c8-478d-ba9d-efd09407cde7" (UID: "de3d0eaa-76c8-478d-ba9d-efd09407cde7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:11:50.885203 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.885177    2585 generic.go:356] "Generic (PLEG): container finished" podID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerID="28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47" exitCode=2
Jul 24 22:11:50.885309 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.885252    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerDied","Data":"28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47"}
Jul 24 22:11:50.886453 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.886436    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz_de3d0eaa-76c8-478d-ba9d-efd09407cde7/storage-initializer/1.log"
Jul 24 22:11:50.886545 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.886500    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz" event={"ID":"de3d0eaa-76c8-478d-ba9d-efd09407cde7","Type":"ContainerDied","Data":"e33c7c863840880a0d6a2fd3cc249b32d087c4f81c2495c200a724d716d8395e"}
Jul 24 22:11:50.886545 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.886533    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"
Jul 24 22:11:50.886618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.886560    2585 scope.go:117] "RemoveContainer" containerID="2bfb65cf2d1b28a903a19a8145fd30ae6ecf7eeddcaf6555db664e042d881e58"
Jul 24 22:11:50.923540 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.923506    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"]
Jul 24 22:11:50.926751 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.926728    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5c6eb3-predictor-7745f9d6b5-hrdgz"]
Jul 24 22:11:50.947851 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.947830    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:50.947851 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.947850    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:50.947958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.947861    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92n8\" (UniqueName: \"kubernetes.io/projected/de3d0eaa-76c8-478d-ba9d-efd09407cde7-kube-api-access-k92n8\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:50.947958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:50.947871    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3d0eaa-76c8-478d-ba9d-efd09407cde7-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:51.326167 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:51.326135    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" path="/var/lib/kubelet/pods/de3d0eaa-76c8-478d-ba9d-efd09407cde7/volumes"
Jul 24 22:11:54.841581 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.841555    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:11:54.902881 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.902845    2585 generic.go:356] "Generic (PLEG): container finished" podID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerID="73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f" exitCode=0
Jul 24 22:11:54.903047 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.902900    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"
Jul 24 22:11:54.903047 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.902916    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerDied","Data":"73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f"}
Jul 24 22:11:54.903047 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.902974    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp" event={"ID":"d273ef49-6601-4b1b-a1fc-19e9bd444c7e","Type":"ContainerDied","Data":"893abbb9da6e793611fdeaa3292bd18344843e94125ac8d8f86127d1e1c943e6"}
Jul 24 22:11:54.903047 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.903002    2585 scope.go:117] "RemoveContainer" containerID="28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47"
Jul 24 22:11:54.910881 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.910865    2585 scope.go:117] "RemoveContainer" containerID="73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f"
Jul 24 22:11:54.917965 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.917948    2585 scope.go:117] "RemoveContainer" containerID="bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73"
Jul 24 22:11:54.927302 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.927277    2585 scope.go:117] "RemoveContainer" containerID="28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47"
Jul 24 22:11:54.927528 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:11:54.927510    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47\": container with ID starting with 28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47 not found: ID does not exist" containerID="28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47"
Jul 24 22:11:54.927586 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.927538    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47"} err="failed to get container status \"28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47\": rpc error: code = NotFound desc = could not find container \"28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47\": container with ID starting with 28e3358fb7d9b609ccb25b33e95b7815f7ed318702725a12f1d4a2583c263d47 not found: ID does not exist"
Jul 24 22:11:54.927586 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.927556    2585 scope.go:117] "RemoveContainer" containerID="73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f"
Jul 24 22:11:54.927832 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:11:54.927815    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f\": container with ID starting with 73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f not found: ID does not exist" containerID="73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f"
Jul 24 22:11:54.927883 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.927839    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f"} err="failed to get container status \"73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f\": rpc error: code = NotFound desc = could not find container \"73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f\": container with ID starting with 73353ad3fb5ec1bc969e4cb7bbbf0d0abc751551ef0d86a918c619421f98873f not found: ID does not exist"
Jul 24 22:11:54.927883 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.927859    2585 scope.go:117] "RemoveContainer" containerID="bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73"
Jul 24 22:11:54.928087 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:11:54.928070    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73\": container with ID starting with bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73 not found: ID does not exist" containerID="bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73"
Jul 24 22:11:54.928138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.928091    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73"} err="failed to get container status \"bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73\": rpc error: code = NotFound desc = could not find container \"bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73\": container with ID starting with bed675ad6db36af00c42f69e94fc3460e004417a10ff15d71b1e7217dc201d73 not found: ID does not exist"
Jul 24 22:11:54.975877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.975832    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls\") pod \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") "
Jul 24 22:11:54.975967 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.975876    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5nhh\" (UniqueName: \"kubernetes.io/projected/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-api-access-s5nhh\") pod \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") "
Jul 24 22:11:54.975967 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.975896    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-rbac-proxy-sar-config\") pod \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") "
Jul 24 22:11:54.975967 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.975914    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-cabundle-cert\") pod \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") "
Jul 24 22:11:54.976128 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.975978    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kserve-provision-location\") pod \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\" (UID: \"d273ef49-6601-4b1b-a1fc-19e9bd444c7e\") "
Jul 24 22:11:54.976312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.976287    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d273ef49-6601-4b1b-a1fc-19e9bd444c7e" (UID: "d273ef49-6601-4b1b-a1fc-19e9bd444c7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:11:54.976312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.976293    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d273ef49-6601-4b1b-a1fc-19e9bd444c7e" (UID: "d273ef49-6601-4b1b-a1fc-19e9bd444c7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:11:54.976423 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.976296    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d273ef49-6601-4b1b-a1fc-19e9bd444c7e" (UID: "d273ef49-6601-4b1b-a1fc-19e9bd444c7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:11:54.978018 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.977994    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d273ef49-6601-4b1b-a1fc-19e9bd444c7e" (UID: "d273ef49-6601-4b1b-a1fc-19e9bd444c7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:11:54.978166 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:54.978145    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-api-access-s5nhh" (OuterVolumeSpecName: "kube-api-access-s5nhh") pod "d273ef49-6601-4b1b-a1fc-19e9bd444c7e" (UID: "d273ef49-6601-4b1b-a1fc-19e9bd444c7e"). InnerVolumeSpecName "kube-api-access-s5nhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:11:55.077121 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.077079    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:55.077121 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.077120    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:55.077285 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.077135    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:55.077285 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.077147    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5nhh\" (UniqueName: \"kubernetes.io/projected/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-api-access-s5nhh\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:55.077285 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.077159    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d273ef49-6601-4b1b-a1fc-19e9bd444c7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:11:55.224855 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.224826    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"]
Jul 24 22:11:55.229466 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.229396    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-5c6eb3-predictor-5587458446-fllrp"]
Jul 24 22:11:55.327013 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.326974    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" path="/var/lib/kubelet/pods/d273ef49-6601-4b1b-a1fc-19e9bd444c7e/volumes"
Jul 24 22:11:55.812441 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.812407    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:11:55.843993 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.843968    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"]
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844226    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844236    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844246    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="storage-initializer"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844252    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="storage-initializer"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844259    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerName="storage-initializer"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844265    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerName="storage-initializer"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844273    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kube-rbac-proxy"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844278    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kube-rbac-proxy"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844283    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerName="storage-initializer"
Jul 24 22:11:55.844323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844288    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerName="storage-initializer"
Jul 24 22:11:55.844647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844332    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerName="storage-initializer"
Jul 24 22:11:55.844647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844340    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3d0eaa-76c8-478d-ba9d-efd09407cde7" containerName="storage-initializer"
Jul 24 22:11:55.844647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844348    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kserve-container"
Jul 24 22:11:55.844647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.844354    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d273ef49-6601-4b1b-a1fc-19e9bd444c7e" containerName="kube-rbac-proxy"
Jul 24 22:11:55.848916 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.848900    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:55.851820 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.851774    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-baf50e\""
Jul 24 22:11:55.851820 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.851793    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-baf50e-kube-rbac-proxy-sar-config\""
Jul 24 22:11:55.852003 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.851896    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:11:55.852003 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.851933    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:11:55.852097 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.852043    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-baf50e-dockercfg-qk755\""
Jul 24 22:11:55.852182 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.852165    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:11:55.852928 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.852908    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-baf50e-predictor-serving-cert\""
Jul 24 22:11:55.858315 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.858295    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"]
Jul 24 22:11:55.984581 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.984550    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:55.984777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.984619    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pph96\" (UniqueName: \"kubernetes.io/projected/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-api-access-pph96\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:55.984777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.984693    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d031e2-b751-4937-91c3-8306cbffa2ee-proxy-tls\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:55.984777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.984736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46d031e2-b751-4937-91c3-8306cbffa2ee-kserve-provision-location\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:55.984777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:55.984772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-cabundle-cert\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.085962 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.085881    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pph96\" (UniqueName: \"kubernetes.io/projected/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-api-access-pph96\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.085962 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.085916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d031e2-b751-4937-91c3-8306cbffa2ee-proxy-tls\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.085962 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.085942    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46d031e2-b751-4937-91c3-8306cbffa2ee-kserve-provision-location\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.086237 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.086067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-cabundle-cert\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.086237 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.086160    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.086389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.086367    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46d031e2-b751-4937-91c3-8306cbffa2ee-kserve-provision-location\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.086899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.086872    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.086899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.086883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-cabundle-cert\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.088593 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.088572    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d031e2-b751-4937-91c3-8306cbffa2ee-proxy-tls\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.093961 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.093940    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pph96\" (UniqueName: \"kubernetes.io/projected/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-api-access-pph96\") pod \"isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") " pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.111514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.111480    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:11:56.160658 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.160636    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:11:56.291625 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.291595    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"]
Jul 24 22:11:56.294791 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:11:56.294769    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d031e2_b751_4937_91c3_8306cbffa2ee.slice/crio-b4f1fafec68f032c25549baba76b033793b5e8e8b56ce523c0344ca53bd5acec WatchSource:0}: Error finding container b4f1fafec68f032c25549baba76b033793b5e8e8b56ce523c0344ca53bd5acec: Status 404 returned error can't find the container with id b4f1fafec68f032c25549baba76b033793b5e8e8b56ce523c0344ca53bd5acec
Jul 24 22:11:56.912734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.912699    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" event={"ID":"46d031e2-b751-4937-91c3-8306cbffa2ee","Type":"ContainerStarted","Data":"cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30"}
Jul 24 22:11:56.913114 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:56.912756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" event={"ID":"46d031e2-b751-4937-91c3-8306cbffa2ee","Type":"ContainerStarted","Data":"b4f1fafec68f032c25549baba76b033793b5e8e8b56ce523c0344ca53bd5acec"}
Jul 24 22:11:58.311774 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:11:58.311740    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:12:01.932688 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:01.932639    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26_46d031e2-b751-4937-91c3-8306cbffa2ee/storage-initializer/0.log"
Jul 24 22:12:01.933059 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:01.932701    2585 generic.go:356] "Generic (PLEG): container finished" podID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerID="cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30" exitCode=1
Jul 24 22:12:01.933059 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:01.932748    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" event={"ID":"46d031e2-b751-4937-91c3-8306cbffa2ee","Type":"ContainerDied","Data":"cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30"}
Jul 24 22:12:02.937707 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:02.937680    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26_46d031e2-b751-4937-91c3-8306cbffa2ee/storage-initializer/0.log"
Jul 24 22:12:02.938077 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:02.937782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" event={"ID":"46d031e2-b751-4937-91c3-8306cbffa2ee","Type":"ContainerStarted","Data":"2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e"}
Jul 24 22:12:05.857097 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:05.857066    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"]
Jul 24 22:12:05.857613 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:05.857262    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer" containerID="cri-o://2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e" gracePeriod=30
Jul 24 22:12:05.871950 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:05.871927    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:12:07.203046 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.203026    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26_46d031e2-b751-4937-91c3-8306cbffa2ee/storage-initializer/1.log"
Jul 24 22:12:07.203388 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.203372    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26_46d031e2-b751-4937-91c3-8306cbffa2ee/storage-initializer/0.log"
Jul 24 22:12:07.205730 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.205715    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:12:07.371573 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.371549    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pph96\" (UniqueName: \"kubernetes.io/projected/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-api-access-pph96\") pod \"46d031e2-b751-4937-91c3-8306cbffa2ee\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") "
Jul 24 22:12:07.371741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.371606    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-rbac-proxy-sar-config\") pod \"46d031e2-b751-4937-91c3-8306cbffa2ee\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") "
Jul 24 22:12:07.371741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.371650    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46d031e2-b751-4937-91c3-8306cbffa2ee-kserve-provision-location\") pod \"46d031e2-b751-4937-91c3-8306cbffa2ee\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") "
Jul 24 22:12:07.371741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.371691    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d031e2-b751-4937-91c3-8306cbffa2ee-proxy-tls\") pod \"46d031e2-b751-4937-91c3-8306cbffa2ee\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") "
Jul 24 22:12:07.371741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.371721    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-cabundle-cert\") pod \"46d031e2-b751-4937-91c3-8306cbffa2ee\" (UID: \"46d031e2-b751-4937-91c3-8306cbffa2ee\") "
Jul 24 22:12:07.372007 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.371982    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d031e2-b751-4937-91c3-8306cbffa2ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46d031e2-b751-4937-91c3-8306cbffa2ee" (UID: "46d031e2-b751-4937-91c3-8306cbffa2ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:12:07.372070 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.372049    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46d031e2-b751-4937-91c3-8306cbffa2ee" (UID: "46d031e2-b751-4937-91c3-8306cbffa2ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:12:07.372176 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.372155    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46d031e2-b751-4937-91c3-8306cbffa2ee" (UID: "46d031e2-b751-4937-91c3-8306cbffa2ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:12:07.373772 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.373753    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-api-access-pph96" (OuterVolumeSpecName: "kube-api-access-pph96") pod "46d031e2-b751-4937-91c3-8306cbffa2ee" (UID: "46d031e2-b751-4937-91c3-8306cbffa2ee"). InnerVolumeSpecName "kube-api-access-pph96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:12:07.373874 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.373824    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d031e2-b751-4937-91c3-8306cbffa2ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46d031e2-b751-4937-91c3-8306cbffa2ee" (UID: "46d031e2-b751-4937-91c3-8306cbffa2ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:12:07.472738 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.472455    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46d031e2-b751-4937-91c3-8306cbffa2ee-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:12:07.472738 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.472481    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d031e2-b751-4937-91c3-8306cbffa2ee-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:12:07.472738 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.472495    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:12:07.472738 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.472507    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pph96\" (UniqueName: \"kubernetes.io/projected/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-api-access-pph96\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:12:07.472738 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.472522    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d031e2-b751-4937-91c3-8306cbffa2ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:12:07.956892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.956864    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26_46d031e2-b751-4937-91c3-8306cbffa2ee/storage-initializer/1.log"
Jul 24 22:12:07.957291 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.957269    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26_46d031e2-b751-4937-91c3-8306cbffa2ee/storage-initializer/0.log"
Jul 24 22:12:07.957408 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.957317    2585 generic.go:356] "Generic (PLEG): container finished" podID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerID="2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e" exitCode=1
Jul 24 22:12:07.957408 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.957376    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"
Jul 24 22:12:07.957516 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.957378    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" event={"ID":"46d031e2-b751-4937-91c3-8306cbffa2ee","Type":"ContainerDied","Data":"2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e"}
Jul 24 22:12:07.957572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.957522    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26" event={"ID":"46d031e2-b751-4937-91c3-8306cbffa2ee","Type":"ContainerDied","Data":"b4f1fafec68f032c25549baba76b033793b5e8e8b56ce523c0344ca53bd5acec"}
Jul 24 22:12:07.957572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.957567    2585 scope.go:117] "RemoveContainer" containerID="2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e"
Jul 24 22:12:07.966888 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.966870    2585 scope.go:117] "RemoveContainer" containerID="cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30"
Jul 24 22:12:07.974353 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.974333    2585 scope.go:117] "RemoveContainer" containerID="2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e"
Jul 24 22:12:07.974623 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:12:07.974604    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e\": container with ID starting with 2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e not found: ID does not exist" containerID="2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e"
Jul 24 22:12:07.974692 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.974632    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e"} err="failed to get container status \"2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e\": rpc error: code = NotFound desc = could not find container \"2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e\": container with ID starting with 2c5fa9229de56f0880044acd32141a1e7e3c2d4fbb819fa54c172bbb2d5e634e not found: ID does not exist"
Jul 24 22:12:07.974692 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.974650    2585 scope.go:117] "RemoveContainer" containerID="cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30"
Jul 24 22:12:07.974934 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:12:07.974915    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30\": container with ID starting with cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30 not found: ID does not exist" containerID="cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30"
Jul 24 22:12:07.974992 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.974942    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30"} err="failed to get container status \"cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30\": rpc error: code = NotFound desc = could not find container \"cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30\": container with ID starting with cb4fff4311f3cda5ae79e7ea193d3762b15ae60d4663f6c44fd88bb34e979d30 not found: ID does not exist"
Jul 24 22:12:07.994260 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.994232    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"]
Jul 24 22:12:07.998305 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:07.998280    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-baf50e-predictor-5dcc4f6f45-7bv26"]
Jul 24 22:12:09.326993 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:09.326960    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" path="/var/lib/kubelet/pods/46d031e2-b751-4937-91c3-8306cbffa2ee/volumes"
Jul 24 22:12:11.081438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.081352    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:12:11.115777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.115745    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"]
Jul 24 22:12:11.116021 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.116009    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer"
Jul 24 22:12:11.116069 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.116023    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer"
Jul 24 22:12:11.116069 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.116035    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer"
Jul 24 22:12:11.116069 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.116041    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer"
Jul 24 22:12:11.116161 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.116084    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer"
Jul 24 22:12:11.116161 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.116092    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d031e2-b751-4937-91c3-8306cbffa2ee" containerName="storage-initializer"
Jul 24 22:12:11.120460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.120440    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.123439 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.123409    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:12:11.123439 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.123422    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 24 22:12:11.123628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.123447    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 24 22:12:11.123628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.123409    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:12:11.123628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.123460    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:12:11.123628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.123429    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:12:11.124807 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.124786    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:12:11.129887 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.129867    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"]
Jul 24 22:12:11.298595 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.298562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhljj\" (UniqueName: \"kubernetes.io/projected/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-api-access-xhljj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.298595 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.298596    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f75dcab-d805-40f1-a969-f5faf42e93ab-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.298857 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.298627    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.298857 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.298757    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f75dcab-d805-40f1-a969-f5faf42e93ab-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.298857 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.298810    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399098 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhljj\" (UniqueName: \"kubernetes.io/projected/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-api-access-xhljj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399110    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f75dcab-d805-40f1-a969-f5faf42e93ab-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399259    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f75dcab-d805-40f1-a969-f5faf42e93ab-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399551 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399532    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f75dcab-d805-40f1-a969-f5faf42e93ab-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399855 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.399958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.399942    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.401811 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.401789    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f75dcab-d805-40f1-a969-f5faf42e93ab-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.422725 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.422696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhljj\" (UniqueName: \"kubernetes.io/projected/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-api-access-xhljj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-z4q42\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.432421 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.432398    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:11.467601 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.467569    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:12:11.581296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.581193    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"]
Jul 24 22:12:11.584174 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:12:11.584140    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f75dcab_d805_40f1_a969_f5faf42e93ab.slice/crio-bdd5dcdaf95ca841dd32895946eaef077d264c55cf840d525a277236a3850662 WatchSource:0}: Error finding container bdd5dcdaf95ca841dd32895946eaef077d264c55cf840d525a277236a3850662: Status 404 returned error can't find the container with id bdd5dcdaf95ca841dd32895946eaef077d264c55cf840d525a277236a3850662
Jul 24 22:12:11.976137 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.976086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerStarted","Data":"f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f"}
Jul 24 22:12:11.976298 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:11.976181    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerStarted","Data":"bdd5dcdaf95ca841dd32895946eaef077d264c55cf840d525a277236a3850662"}
Jul 24 22:12:15.993486 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:15.993448    2585 generic.go:356] "Generic (PLEG): container finished" podID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerID="f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f" exitCode=0
Jul 24 22:12:15.993949 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:15.993528    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerDied","Data":"f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f"}
Jul 24 22:12:15.994890 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:15.994873    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:12:37.096050 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:37.096012    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerStarted","Data":"6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c"}
Jul 24 22:12:37.096478 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:37.096100    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerStarted","Data":"81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650"}
Jul 24 22:12:37.096478 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:37.096379    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:37.117364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:37.117278    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podStartSLOduration=1.5787277720000001 podStartE2EDuration="26.117261341s" totalImagesPullingTime="20.861616846s" totalInitContainerRuntime="3.676916723s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:12:37.115382454 +0000 UTC m=+3438.308560415" watchObservedRunningTime="2026-07-24 22:12:37.117261341 +0000 UTC m=+3438.310439278"
Jul 24 22:12:38.100025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:38.099989    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:38.101094 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:38.101070    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:12:39.103611 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:39.103571    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:12:44.108062 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:44.108029    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:12:44.108536 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:44.108510    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:12:54.109486 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:12:54.109445    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:13:04.109033 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:13:04.108991    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:13:14.108556 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:13:14.108512    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:13:24.108449 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:13:24.108407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:13:34.108814 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:13:34.108768    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:13:44.108587 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:13:44.108486    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:13:54.109486 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:13:54.109448    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:14:01.251602 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:01.251570    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"]
Jul 24 22:14:01.252069 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:01.251836    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" containerID="cri-o://81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650" gracePeriod=30
Jul 24 22:14:01.252069 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:01.251890    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kube-rbac-proxy" containerID="cri-o://6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c" gracePeriod=30
Jul 24 22:14:01.305202 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:01.305169    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:14:01.411017 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:01.410991    2585 generic.go:356] "Generic (PLEG): container finished" podID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerID="6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c" exitCode=2
Jul 24 22:14:01.411149 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:01.411071    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerDied","Data":"6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c"}
Jul 24 22:14:04.104245 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:04.104200    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 24 22:14:04.108426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:04.108401    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 24 22:14:05.590834 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.590811    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:14:05.676910 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.676849    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhljj\" (UniqueName: \"kubernetes.io/projected/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-api-access-xhljj\") pod \"0f75dcab-d805-40f1-a969-f5faf42e93ab\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") "
Jul 24 22:14:05.676910 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.676884    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f75dcab-d805-40f1-a969-f5faf42e93ab-kserve-provision-location\") pod \"0f75dcab-d805-40f1-a969-f5faf42e93ab\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") "
Jul 24 22:14:05.677063 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.676913    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f75dcab-d805-40f1-a969-f5faf42e93ab-proxy-tls\") pod \"0f75dcab-d805-40f1-a969-f5faf42e93ab\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") "
Jul 24 22:14:05.677063 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.676937    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-rbac-proxy-sar-config\") pod \"0f75dcab-d805-40f1-a969-f5faf42e93ab\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") "
Jul 24 22:14:05.677063 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.676966    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-cabundle-cert\") pod \"0f75dcab-d805-40f1-a969-f5faf42e93ab\" (UID: \"0f75dcab-d805-40f1-a969-f5faf42e93ab\") "
Jul 24 22:14:05.677247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.677219    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f75dcab-d805-40f1-a969-f5faf42e93ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f75dcab-d805-40f1-a969-f5faf42e93ab" (UID: "0f75dcab-d805-40f1-a969-f5faf42e93ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:14:05.677321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.677301    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f75dcab-d805-40f1-a969-f5faf42e93ab" (UID: "0f75dcab-d805-40f1-a969-f5faf42e93ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:14:05.677357 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.677308    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f75dcab-d805-40f1-a969-f5faf42e93ab" (UID: "0f75dcab-d805-40f1-a969-f5faf42e93ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:14:05.679151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.679131    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f75dcab-d805-40f1-a969-f5faf42e93ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f75dcab-d805-40f1-a969-f5faf42e93ab" (UID: "0f75dcab-d805-40f1-a969-f5faf42e93ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:14:05.679213 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.679172    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-api-access-xhljj" (OuterVolumeSpecName: "kube-api-access-xhljj") pod "0f75dcab-d805-40f1-a969-f5faf42e93ab" (UID: "0f75dcab-d805-40f1-a969-f5faf42e93ab"). InnerVolumeSpecName "kube-api-access-xhljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:14:05.777822 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.777801    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f75dcab-d805-40f1-a969-f5faf42e93ab-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:14:05.777822 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.777821    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:14:05.777968 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.777831    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f75dcab-d805-40f1-a969-f5faf42e93ab-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:14:05.777968 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.777841    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhljj\" (UniqueName: \"kubernetes.io/projected/0f75dcab-d805-40f1-a969-f5faf42e93ab-kube-api-access-xhljj\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:14:05.777968 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:05.777851    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f75dcab-d805-40f1-a969-f5faf42e93ab-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:14:06.354515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.354484    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:14:06.430381 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.430348    2585 generic.go:356] "Generic (PLEG): container finished" podID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerID="81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650" exitCode=0
Jul 24 22:14:06.430542 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.430394    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerDied","Data":"81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650"}
Jul 24 22:14:06.430542 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.430408    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"
Jul 24 22:14:06.430542 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.430443    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42" event={"ID":"0f75dcab-d805-40f1-a969-f5faf42e93ab","Type":"ContainerDied","Data":"bdd5dcdaf95ca841dd32895946eaef077d264c55cf840d525a277236a3850662"}
Jul 24 22:14:06.430542 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.430469    2585 scope.go:117] "RemoveContainer" containerID="6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c"
Jul 24 22:14:06.440003 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.439982    2585 scope.go:117] "RemoveContainer" containerID="81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650"
Jul 24 22:14:06.447306 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.447286    2585 scope.go:117] "RemoveContainer" containerID="f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f"
Jul 24 22:14:06.453374 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453352    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"]
Jul 24 22:14:06.453734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453718    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="storage-initializer"
Jul 24 22:14:06.453812 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453736    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="storage-initializer"
Jul 24 22:14:06.453812 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453754    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kube-rbac-proxy"
Jul 24 22:14:06.453812 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453763    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kube-rbac-proxy"
Jul 24 22:14:06.453812 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453772    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container"
Jul 24 22:14:06.453812 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453780    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container"
Jul 24 22:14:06.453986 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453856    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kserve-container"
Jul 24 22:14:06.453986 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.453865    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" containerName="kube-rbac-proxy"
Jul 24 22:14:06.457722 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.455372    2585 scope.go:117] "RemoveContainer" containerID="6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c"
Jul 24 22:14:06.462821 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:14:06.462786    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c\": container with ID starting with 6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c not found: ID does not exist" containerID="6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c"
Jul 24 22:14:06.462994 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.462974    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c"} err="failed to get container status \"6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c\": rpc error: code = NotFound desc = could not find container \"6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c\": container with ID starting with 6ddaabce51be658556e5893a8e2eb53998b49f7642aad473f9fbd0a6eb97343c not found: ID does not exist"
Jul 24 22:14:06.463044 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.462996    2585 scope.go:117] "RemoveContainer" containerID="81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650"
Jul 24 22:14:06.463413 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:14:06.463384    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650\": container with ID starting with 81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650 not found: ID does not exist" containerID="81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650"
Jul 24 22:14:06.463473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.463426    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650"} err="failed to get container status \"81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650\": rpc error: code = NotFound desc = could not find container \"81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650\": container with ID starting with 81ea825a2d045ee2a615ca3bb6c5e20376387b5f790e49ccf84b4f6dbb833650 not found: ID does not exist"
Jul 24 22:14:06.463473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.463449    2585 scope.go:117] "RemoveContainer" containerID="f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f"
Jul 24 22:14:06.463787 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:14:06.463763    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f\": container with ID starting with f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f not found: ID does not exist" containerID="f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f"
Jul 24 22:14:06.463870 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.463796    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f"} err="failed to get container status \"f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f\": rpc error: code = NotFound desc = could not find container \"f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f\": container with ID starting with f863951ef14e4aec9432e1241b8caa54bcc1709e112b0f0f01d367155dad3d3f not found: ID does not exist"
Jul 24 22:14:06.463996 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.463978    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"]
Jul 24 22:14:06.464102 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.464086    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.464607 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.464584    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-z4q42"]
Jul 24 22:14:06.467202 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.467183    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 24 22:14:06.467352 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.467335    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:14:06.467430 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.467258    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:14:06.467430 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.467250    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:14:06.467640 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.467624    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:14:06.467947 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.467924    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 24 22:14:06.468046 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.468008    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:14:06.468766 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.468745    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"]
Jul 24 22:14:06.583461 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.583436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.583579 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.583467    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.583579 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.583491    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.583579 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.583533    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.583579 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.583553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfz25\" (UniqueName: \"kubernetes.io/projected/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-api-access-nfz25\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.651937 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.651905    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:14:06.684036 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684011    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684044    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684107    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684138    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfz25\" (UniqueName: \"kubernetes.io/projected/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-api-access-nfz25\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684576 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684807 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.684867 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.684839    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.686531 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.686507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.693253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.693232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfz25\" (UniqueName: \"kubernetes.io/projected/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-api-access-nfz25\") pod \"isvc-predictive-xgboost-predictor-695f978978-64tnq\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.776206 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.776183    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:06.901861 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:06.901835    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"]
Jul 24 22:14:06.904126 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:14:06.904065    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ac5558_ff94_4fa7_8643_94b9b7649cd6.slice/crio-a4444b72dd5174e5e61b27a87b1f77136c8653f00682c06285eac4aadc3104e6 WatchSource:0}: Error finding container a4444b72dd5174e5e61b27a87b1f77136c8653f00682c06285eac4aadc3104e6: Status 404 returned error can't find the container with id a4444b72dd5174e5e61b27a87b1f77136c8653f00682c06285eac4aadc3104e6
Jul 24 22:14:07.326461 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:07.326384    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f75dcab-d805-40f1-a969-f5faf42e93ab" path="/var/lib/kubelet/pods/0f75dcab-d805-40f1-a969-f5faf42e93ab/volumes"
Jul 24 22:14:07.436301 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:07.436260    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerStarted","Data":"cbc7adb76f6e80b93eafc6aee6a8fe76cce6b8da5472b12f384eb75a2634629a"}
Jul 24 22:14:07.436479 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:07.436336    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerStarted","Data":"a4444b72dd5174e5e61b27a87b1f77136c8653f00682c06285eac4aadc3104e6"}
Jul 24 22:14:11.453724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:11.453691    2585 generic.go:356] "Generic (PLEG): container finished" podID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerID="cbc7adb76f6e80b93eafc6aee6a8fe76cce6b8da5472b12f384eb75a2634629a" exitCode=0
Jul 24 22:14:11.454108 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:11.453764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerDied","Data":"cbc7adb76f6e80b93eafc6aee6a8fe76cce6b8da5472b12f384eb75a2634629a"}
Jul 24 22:14:12.460053 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:12.460018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerStarted","Data":"f1c18571c1f2d366114e4cb1fb6d723eaabb3b65c97fe5ab428f67bf070bd61a"}
Jul 24 22:14:12.460515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:12.460092    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerStarted","Data":"fc449da0c16ac63c68e97ba1a8fed0e5bf286150d001bc12b09003dcc5d8a54f"}
Jul 24 22:14:12.460515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:12.460314    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:12.479494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:12.479444    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podStartSLOduration=2.9501965180000003 podStartE2EDuration="6.479427967s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.529231449s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:14:12.47847135 +0000 UTC m=+3533.671649315" watchObservedRunningTime="2026-07-24 22:14:12.479427967 +0000 UTC m=+3533.672605905"
Jul 24 22:14:13.466980 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:13.466946    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:13.468142 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:13.468114    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:14:14.470569 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:14.470524    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:14:19.475325 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:19.475298    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:14:19.475843 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:19.475815    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:14:29.476241 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:29.476195    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:14:39.475911 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:39.475872    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:14:49.476044 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:49.476007    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:14:59.475985 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:14:59.475944    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:15:09.475899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:09.475854    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:15:19.476847 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:19.476818    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:15:26.528339 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:26.528307    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"]
Jul 24 22:15:26.528803 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:26.528555    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" containerID="cri-o://fc449da0c16ac63c68e97ba1a8fed0e5bf286150d001bc12b09003dcc5d8a54f" gracePeriod=30
Jul 24 22:15:26.528803 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:26.528599    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kube-rbac-proxy" containerID="cri-o://f1c18571c1f2d366114e4cb1fb6d723eaabb3b65c97fe5ab428f67bf070bd61a" gracePeriod=30
Jul 24 22:15:26.545446 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:26.545423    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:15:26.734236 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:26.734204    2585 generic.go:356] "Generic (PLEG): container finished" podID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerID="f1c18571c1f2d366114e4cb1fb6d723eaabb3b65c97fe5ab428f67bf070bd61a" exitCode=2
Jul 24 22:15:26.734366 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:26.734249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerDied","Data":"f1c18571c1f2d366114e4cb1fb6d723eaabb3b65c97fe5ab428f67bf070bd61a"}
Jul 24 22:15:29.471407 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:29.471366    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 24 22:15:29.476482 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:29.476451    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 24 22:15:30.751408 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.751378    2585 generic.go:356] "Generic (PLEG): container finished" podID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerID="fc449da0c16ac63c68e97ba1a8fed0e5bf286150d001bc12b09003dcc5d8a54f" exitCode=0
Jul 24 22:15:30.751867 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.751435    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerDied","Data":"fc449da0c16ac63c68e97ba1a8fed0e5bf286150d001bc12b09003dcc5d8a54f"}
Jul 24 22:15:30.879390 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.879368    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:15:30.967426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967360    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfz25\" (UniqueName: \"kubernetes.io/projected/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-api-access-nfz25\") pod \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") "
Jul 24 22:15:30.967426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kserve-provision-location\") pod \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") "
Jul 24 22:15:30.967426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967421    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-proxy-tls\") pod \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") "
Jul 24 22:15:30.967647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967504    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-cabundle-cert\") pod \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") "
Jul 24 22:15:30.967647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967572    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-rbac-proxy-sar-config\") pod \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\" (UID: \"d3ac5558-ff94-4fa7-8643-94b9b7649cd6\") "
Jul 24 22:15:30.967787 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967762    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3ac5558-ff94-4fa7-8643-94b9b7649cd6" (UID: "d3ac5558-ff94-4fa7-8643-94b9b7649cd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:15:30.967879 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967852    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3ac5558-ff94-4fa7-8643-94b9b7649cd6" (UID: "d3ac5558-ff94-4fa7-8643-94b9b7649cd6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:15:30.968005 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.967981    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3ac5558-ff94-4fa7-8643-94b9b7649cd6" (UID: "d3ac5558-ff94-4fa7-8643-94b9b7649cd6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:15:30.969629 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.969599    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3ac5558-ff94-4fa7-8643-94b9b7649cd6" (UID: "d3ac5558-ff94-4fa7-8643-94b9b7649cd6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:15:30.969629 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:30.969624    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-api-access-nfz25" (OuterVolumeSpecName: "kube-api-access-nfz25") pod "d3ac5558-ff94-4fa7-8643-94b9b7649cd6" (UID: "d3ac5558-ff94-4fa7-8643-94b9b7649cd6"). InnerVolumeSpecName "kube-api-access-nfz25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:15:31.068072 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.068047    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:15:31.068072 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.068070    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:15:31.068222 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.068080    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfz25\" (UniqueName: \"kubernetes.io/projected/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kube-api-access-nfz25\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:15:31.068222 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.068089    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:15:31.068222 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.068099    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3ac5558-ff94-4fa7-8643-94b9b7649cd6-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:15:31.647381 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.647324    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:15:31.682013 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.681981    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"]
Jul 24 22:15:31.682375 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682354    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="storage-initializer"
Jul 24 22:15:31.682375 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682372    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="storage-initializer"
Jul 24 22:15:31.682535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682389    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container"
Jul 24 22:15:31.682535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682397    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container"
Jul 24 22:15:31.682535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682425    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kube-rbac-proxy"
Jul 24 22:15:31.682535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682433    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kube-rbac-proxy"
Jul 24 22:15:31.682535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682491    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kube-rbac-proxy"
Jul 24 22:15:31.682535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.682505    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" containerName="kserve-container"
Jul 24 22:15:31.685888 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.685868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.688459 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.688433    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 24 22:15:31.688616 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.688482    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 24 22:15:31.694892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.694867    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"]
Jul 24 22:15:31.756974 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.756945    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"
Jul 24 22:15:31.757363 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.756944    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq" event={"ID":"d3ac5558-ff94-4fa7-8643-94b9b7649cd6","Type":"ContainerDied","Data":"a4444b72dd5174e5e61b27a87b1f77136c8653f00682c06285eac4aadc3104e6"}
Jul 24 22:15:31.757363 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.757096    2585 scope.go:117] "RemoveContainer" containerID="f1c18571c1f2d366114e4cb1fb6d723eaabb3b65c97fe5ab428f67bf070bd61a"
Jul 24 22:15:31.765726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.765701    2585 scope.go:117] "RemoveContainer" containerID="fc449da0c16ac63c68e97ba1a8fed0e5bf286150d001bc12b09003dcc5d8a54f"
Jul 24 22:15:31.773424 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.773402    2585 scope.go:117] "RemoveContainer" containerID="cbc7adb76f6e80b93eafc6aee6a8fe76cce6b8da5472b12f384eb75a2634629a"
Jul 24 22:15:31.773826 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.773785    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.774009 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.773823    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.774009 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.773865    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d58d2d3-5cf9-4e60-9970-154292a6f231-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.774009 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.773902    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d58d2d3-5cf9-4e60-9970-154292a6f231-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.774009 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.773942    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvwr\" (UniqueName: \"kubernetes.io/projected/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-api-access-jcvwr\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.782527 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.782505    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"]
Jul 24 22:15:31.786589 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.786568    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-64tnq"]
Jul 24 22:15:31.809132 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.809112    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:15:31.874287 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.874428 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.874428 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874383    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d58d2d3-5cf9-4e60-9970-154292a6f231-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.874428 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d58d2d3-5cf9-4e60-9970-154292a6f231-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.874581 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvwr\" (UniqueName: \"kubernetes.io/projected/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-api-access-jcvwr\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.874866 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d58d2d3-5cf9-4e60-9970-154292a6f231-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.874997 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.874981    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.875040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.875016    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.877171 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.877148    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d58d2d3-5cf9-4e60-9970-154292a6f231-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.883865 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.883842    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvwr\" (UniqueName: \"kubernetes.io/projected/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-api-access-jcvwr\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:31.998079 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:31.998006    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:32.129257 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:32.129224    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"]
Jul 24 22:15:32.132035 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:15:32.132009    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d58d2d3_5cf9_4e60_9970_154292a6f231.slice/crio-ec5a973beb90af8148edbfdd215ae3ebb3d7a773d563c8c6e39b4f799f907902 WatchSource:0}: Error finding container ec5a973beb90af8148edbfdd215ae3ebb3d7a773d563c8c6e39b4f799f907902: Status 404 returned error can't find the container with id ec5a973beb90af8148edbfdd215ae3ebb3d7a773d563c8c6e39b4f799f907902
Jul 24 22:15:32.763287 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:32.763249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerStarted","Data":"0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7"}
Jul 24 22:15:32.763686 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:32.763314    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerStarted","Data":"ec5a973beb90af8148edbfdd215ae3ebb3d7a773d563c8c6e39b4f799f907902"}
Jul 24 22:15:33.326575 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:33.326545    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ac5558-ff94-4fa7-8643-94b9b7649cd6" path="/var/lib/kubelet/pods/d3ac5558-ff94-4fa7-8643-94b9b7649cd6/volumes"
Jul 24 22:15:36.778996 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:36.778965    2585 generic.go:356] "Generic (PLEG): container finished" podID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerID="0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7" exitCode=0
Jul 24 22:15:36.778996 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:36.779002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerDied","Data":"0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7"}
Jul 24 22:15:37.786051 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:37.786013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerStarted","Data":"316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72"}
Jul 24 22:15:37.786529 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:37.786105    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerStarted","Data":"a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a"}
Jul 24 22:15:37.786529 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:37.786309    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:37.807862 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:37.807816    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podStartSLOduration=3.029481681 podStartE2EDuration="6.8078043s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.778322619s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:15:37.806501034 +0000 UTC m=+3618.999679025" watchObservedRunningTime="2026-07-24 22:15:37.8078043 +0000 UTC m=+3619.000982236"
Jul 24 22:15:38.790248 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:38.790214    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:38.791331 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:38.791306    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:15:39.794848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:39.794807    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:15:44.799036 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:44.799004    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:15:44.799494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:44.799464    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:15:54.800341 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:15:54.800304    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:16:04.799894 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:16:04.799857    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:16:14.799593 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:16:14.799555    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:16:24.799520 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:16:24.799483    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:16:34.800295 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:16:34.800252    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:16:44.799780 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:16:44.799702    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:16:54.800498 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:16:54.800467    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:17:01.841095 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:01.841065    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"]
Jul 24 22:17:01.841558 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:01.841409    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" containerID="cri-o://a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a" gracePeriod=30
Jul 24 22:17:01.841558 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:01.841466    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kube-rbac-proxy" containerID="cri-o://316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72" gracePeriod=30
Jul 24 22:17:01.885150 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:01.885127    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:17:02.096359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:02.096290    2585 generic.go:356] "Generic (PLEG): container finished" podID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerID="316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72" exitCode=2
Jul 24 22:17:02.096359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:02.096353    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerDied","Data":"316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72"}
Jul 24 22:17:04.796037 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:04.795996    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 24 22:17:04.800274 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:04.800251    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 24 22:17:06.947940 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:06.947904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:17:07.048405 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.048356    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:17:07.493263 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.493239    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:17:07.559359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559294    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-rbac-proxy-sar-config\") pod \"8d58d2d3-5cf9-4e60-9970-154292a6f231\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") "
Jul 24 22:17:07.559359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559340    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d58d2d3-5cf9-4e60-9970-154292a6f231-proxy-tls\") pod \"8d58d2d3-5cf9-4e60-9970-154292a6f231\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") "
Jul 24 22:17:07.559359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559358    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d58d2d3-5cf9-4e60-9970-154292a6f231-kserve-provision-location\") pod \"8d58d2d3-5cf9-4e60-9970-154292a6f231\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") "
Jul 24 22:17:07.559564 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-cabundle-cert\") pod \"8d58d2d3-5cf9-4e60-9970-154292a6f231\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") "
Jul 24 22:17:07.559564 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559435    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcvwr\" (UniqueName: \"kubernetes.io/projected/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-api-access-jcvwr\") pod \"8d58d2d3-5cf9-4e60-9970-154292a6f231\" (UID: \"8d58d2d3-5cf9-4e60-9970-154292a6f231\") "
Jul 24 22:17:07.559783 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559748    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d58d2d3-5cf9-4e60-9970-154292a6f231" (UID: "8d58d2d3-5cf9-4e60-9970-154292a6f231"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:17:07.559884 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559778    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d58d2d3-5cf9-4e60-9970-154292a6f231-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d58d2d3-5cf9-4e60-9970-154292a6f231" (UID: "8d58d2d3-5cf9-4e60-9970-154292a6f231"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:17:07.559884 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.559810    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d58d2d3-5cf9-4e60-9970-154292a6f231" (UID: "8d58d2d3-5cf9-4e60-9970-154292a6f231"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:17:07.561495 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.561479    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d58d2d3-5cf9-4e60-9970-154292a6f231-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d58d2d3-5cf9-4e60-9970-154292a6f231" (UID: "8d58d2d3-5cf9-4e60-9970-154292a6f231"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:17:07.561727 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.561704    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-api-access-jcvwr" (OuterVolumeSpecName: "kube-api-access-jcvwr") pod "8d58d2d3-5cf9-4e60-9970-154292a6f231" (UID: "8d58d2d3-5cf9-4e60-9970-154292a6f231"). InnerVolumeSpecName "kube-api-access-jcvwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:17:07.660719 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.660698    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvwr\" (UniqueName: \"kubernetes.io/projected/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-api-access-jcvwr\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:17:07.660719 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.660718    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:17:07.660860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.660728    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d58d2d3-5cf9-4e60-9970-154292a6f231-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:17:07.660860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.660736    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d58d2d3-5cf9-4e60-9970-154292a6f231-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:17:07.660860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:07.660745    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d58d2d3-5cf9-4e60-9970-154292a6f231-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:17:08.119321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.119284    2585 generic.go:356] "Generic (PLEG): container finished" podID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerID="a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a" exitCode=0
Jul 24 22:17:08.119726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.119328    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerDied","Data":"a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a"}
Jul 24 22:17:08.119726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.119350    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"
Jul 24 22:17:08.119726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.119390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz" event={"ID":"8d58d2d3-5cf9-4e60-9970-154292a6f231","Type":"ContainerDied","Data":"ec5a973beb90af8148edbfdd215ae3ebb3d7a773d563c8c6e39b4f799f907902"}
Jul 24 22:17:08.119726 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.119432    2585 scope.go:117] "RemoveContainer" containerID="316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72"
Jul 24 22:17:08.127880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.127860    2585 scope.go:117] "RemoveContainer" containerID="a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a"
Jul 24 22:17:08.135045 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.135028    2585 scope.go:117] "RemoveContainer" containerID="0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7"
Jul 24 22:17:08.141990 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.141969    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"]
Jul 24 22:17:08.142485 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.142472    2585 scope.go:117] "RemoveContainer" containerID="316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72"
Jul 24 22:17:08.142828 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:17:08.142805    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72\": container with ID starting with 316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72 not found: ID does not exist" containerID="316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72"
Jul 24 22:17:08.142899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.142836    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72"} err="failed to get container status \"316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72\": rpc error: code = NotFound desc = could not find container \"316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72\": container with ID starting with 316bd9e170790ab6d6134d6f30ed639d2d1c38cae80d9069b83e43f39974dd72 not found: ID does not exist"
Jul 24 22:17:08.142899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.142854    2585 scope.go:117] "RemoveContainer" containerID="a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a"
Jul 24 22:17:08.143102 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:17:08.143082    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a\": container with ID starting with a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a not found: ID does not exist" containerID="a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a"
Jul 24 22:17:08.143172 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.143113    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a"} err="failed to get container status \"a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a\": rpc error: code = NotFound desc = could not find container \"a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a\": container with ID starting with a621b197989b080e2de24b7d3e0a7c027f3d7ae44c59dffe09dc88a3dd1fc30a not found: ID does not exist"
Jul 24 22:17:08.143172 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.143135    2585 scope.go:117] "RemoveContainer" containerID="0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7"
Jul 24 22:17:08.143394 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:17:08.143376    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7\": container with ID starting with 0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7 not found: ID does not exist" containerID="0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7"
Jul 24 22:17:08.143432 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.143400    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7"} err="failed to get container status \"0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7\": rpc error: code = NotFound desc = could not find container \"0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7\": container with ID starting with 0c8c4acc9ad614b65c8f618b3fcbc66575fd79baabfdb8b424b11790567b53b7 not found: ID does not exist"
Jul 24 22:17:08.146506 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:08.146487    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-kgszz"]
Jul 24 22:17:09.327134 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:17:09.327103    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" path="/var/lib/kubelet/pods/8d58d2d3-5cf9-4e60-9970-154292a6f231/volumes"
Jul 24 22:19:07.235634 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:07.235591    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:19:07.310801 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:07.310775    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:19:12.276830 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.276797    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:19:12.340872 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.340829    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"]
Jul 24 22:19:12.341179 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341163    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="storage-initializer"
Jul 24 22:19:12.341256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341182    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="storage-initializer"
Jul 24 22:19:12.341256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341199    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container"
Jul 24 22:19:12.341256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341205    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container"
Jul 24 22:19:12.341256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341210    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kube-rbac-proxy"
Jul 24 22:19:12.341256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341216    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kube-rbac-proxy"
Jul 24 22:19:12.341508 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341296    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kube-rbac-proxy"
Jul 24 22:19:12.341508 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.341311    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d58d2d3-5cf9-4e60-9970-154292a6f231" containerName="kserve-container"
Jul 24 22:19:12.344562 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.344542    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.347144 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.347113    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:19:12.347284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.347163    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:19:12.347284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.347207    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:19:12.347284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.347246    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:19:12.347450 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.347367    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:19:12.348241 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.348205    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 24 22:19:12.348241 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.348213    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 24 22:19:12.353872 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.353853    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"]
Jul 24 22:19:12.417384 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.417357    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.417513 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.417389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6cz2\" (UniqueName: \"kubernetes.io/projected/c746bf1f-fe37-447b-a830-317e67295a87-kube-api-access-h6cz2\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.417513 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.417410    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c746bf1f-fe37-447b-a830-317e67295a87-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.417513 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.417506    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.417648 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.417536    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.518214 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.518182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.518342 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.518225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.518342 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.518273    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.518342 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.518308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6cz2\" (UniqueName: \"kubernetes.io/projected/c746bf1f-fe37-447b-a830-317e67295a87-kube-api-access-h6cz2\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.518500 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.518344    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c746bf1f-fe37-447b-a830-317e67295a87-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.518500 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:19:12.518456    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 24 22:19:12.518609 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:19:12.518539    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls podName:c746bf1f-fe37-447b-a830-317e67295a87 nodeName:}" failed. No retries permitted until 2026-07-24 22:19:13.018518892 +0000 UTC m=+3834.211696812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" (UID: "c746bf1f-fe37-447b-a830-317e67295a87") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 24 22:19:12.518773 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.518752    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c746bf1f-fe37-447b-a830-317e67295a87-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.519032 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.519010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.519075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.519010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.529237 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.529183    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6cz2\" (UniqueName: \"kubernetes.io/projected/c746bf1f-fe37-447b-a830-317e67295a87-kube-api-access-h6cz2\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:12.618537 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:12.618512    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:19:13.021725 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:13.021697    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:13.024186 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:13.024163    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:13.256121 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:13.256094    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:13.386536 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:13.386510    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"]
Jul 24 22:19:13.388922 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:19:13.388890    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc746bf1f_fe37_447b_a830_317e67295a87.slice/crio-44db7ab26fc243a4734b738127cafb9144df46c12b55d5f313bd8e40d47ada0b WatchSource:0}: Error finding container 44db7ab26fc243a4734b738127cafb9144df46c12b55d5f313bd8e40d47ada0b: Status 404 returned error can't find the container with id 44db7ab26fc243a4734b738127cafb9144df46c12b55d5f313bd8e40d47ada0b
Jul 24 22:19:13.569951 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:13.569870    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerStarted","Data":"58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68"}
Jul 24 22:19:13.569951 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:13.569931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerStarted","Data":"44db7ab26fc243a4734b738127cafb9144df46c12b55d5f313bd8e40d47ada0b"}
Jul 24 22:19:17.585768 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:17.585683    2585 generic.go:356] "Generic (PLEG): container finished" podID="c746bf1f-fe37-447b-a830-317e67295a87" containerID="58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68" exitCode=0
Jul 24 22:19:17.585768 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:17.585735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerDied","Data":"58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68"}
Jul 24 22:19:18.591169 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:18.591138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerStarted","Data":"39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4"}
Jul 24 22:19:18.591519 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:18.591200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerStarted","Data":"ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4"}
Jul 24 22:19:18.591519 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:18.591398    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:18.591612 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:18.591564    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:18.613418 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:18.613374    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podStartSLOduration=2.9479193759999998 podStartE2EDuration="6.613360568s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.665441192s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:19:18.612044076 +0000 UTC m=+3839.805222061" watchObservedRunningTime="2026-07-24 22:19:18.613360568 +0000 UTC m=+3839.806538505"
Jul 24 22:19:24.600061 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:24.600033    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:19:54.601080 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:19:54.600994    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 24 22:20:04.601285 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:04.601236    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 24 22:20:14.601226 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:14.601186    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 24 22:20:24.600560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:24.600521    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 24 22:20:34.685113 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:34.685080    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:20:42.455789 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:42.455756    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"]
Jul 24 22:20:42.456207 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:42.455991    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container" containerID="cri-o://ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4" gracePeriod=30
Jul 24 22:20:42.456207 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:42.456034    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kube-rbac-proxy" containerID="cri-o://39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4" gracePeriod=30
Jul 24 22:20:42.479965 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:42.479937    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:20:42.900955 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:42.900926    2585 generic.go:356] "Generic (PLEG): container finished" podID="c746bf1f-fe37-447b-a830-317e67295a87" containerID="39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4" exitCode=2
Jul 24 22:20:42.901127 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:42.900975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerDied","Data":"39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4"}
Jul 24 22:20:44.595985 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:44.595944    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 24 22:20:44.601431 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:44.601405    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 24 22:20:46.905571 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.905543    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:20:46.916086 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.916062    2585 generic.go:356] "Generic (PLEG): container finished" podID="c746bf1f-fe37-447b-a830-317e67295a87" containerID="ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4" exitCode=0
Jul 24 22:20:46.916208 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.916099    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerDied","Data":"ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4"}
Jul 24 22:20:46.916208 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.916124    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"
Jul 24 22:20:46.916208 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.916145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c" event={"ID":"c746bf1f-fe37-447b-a830-317e67295a87","Type":"ContainerDied","Data":"44db7ab26fc243a4734b738127cafb9144df46c12b55d5f313bd8e40d47ada0b"}
Jul 24 22:20:46.916208 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.916171    2585 scope.go:117] "RemoveContainer" containerID="39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4"
Jul 24 22:20:46.923605 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.923588    2585 scope.go:117] "RemoveContainer" containerID="ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4"
Jul 24 22:20:46.932749 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.932731    2585 scope.go:117] "RemoveContainer" containerID="58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68"
Jul 24 22:20:46.939940 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.939917    2585 scope.go:117] "RemoveContainer" containerID="39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4"
Jul 24 22:20:46.940210 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:20:46.940187    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4\": container with ID starting with 39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4 not found: ID does not exist" containerID="39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4"
Jul 24 22:20:46.940265 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.940223    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4"} err="failed to get container status \"39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4\": rpc error: code = NotFound desc = could not find container \"39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4\": container with ID starting with 39053bba6a8257e95c20d61b197cc89a48e1a0793f85fc9a4955c77d261e78e4 not found: ID does not exist"
Jul 24 22:20:46.940265 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.940246    2585 scope.go:117] "RemoveContainer" containerID="ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4"
Jul 24 22:20:46.940471 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:20:46.940453    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4\": container with ID starting with ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4 not found: ID does not exist" containerID="ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4"
Jul 24 22:20:46.940514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.940480    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4"} err="failed to get container status \"ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4\": rpc error: code = NotFound desc = could not find container \"ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4\": container with ID starting with ba6552028e50548340f32b5236743a281db6cb802eb1897da6fa1b43716874e4 not found: ID does not exist"
Jul 24 22:20:46.940514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.940497    2585 scope.go:117] "RemoveContainer" containerID="58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68"
Jul 24 22:20:46.940788 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:20:46.940771    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68\": container with ID starting with 58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68 not found: ID does not exist" containerID="58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68"
Jul 24 22:20:46.940846 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.940794    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68"} err="failed to get container status \"58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68\": rpc error: code = NotFound desc = could not find container \"58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68\": container with ID starting with 58ba2b7e6fda5367b2c2642da84640eb7edf79302a39cd06daae26dea0922a68 not found: ID does not exist"
Jul 24 22:20:46.995216 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995189    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-cabundle-cert\") pod \"c746bf1f-fe37-447b-a830-317e67295a87\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") "
Jul 24 22:20:46.995314 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995234    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-kube-rbac-proxy-sar-config\") pod \"c746bf1f-fe37-447b-a830-317e67295a87\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") "
Jul 24 22:20:46.995314 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995269    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c746bf1f-fe37-447b-a830-317e67295a87-kserve-provision-location\") pod \"c746bf1f-fe37-447b-a830-317e67295a87\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") "
Jul 24 22:20:46.995314 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995299    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls\") pod \"c746bf1f-fe37-447b-a830-317e67295a87\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") "
Jul 24 22:20:46.995472 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995321    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6cz2\" (UniqueName: \"kubernetes.io/projected/c746bf1f-fe37-447b-a830-317e67295a87-kube-api-access-h6cz2\") pod \"c746bf1f-fe37-447b-a830-317e67295a87\" (UID: \"c746bf1f-fe37-447b-a830-317e67295a87\") "
Jul 24 22:20:46.995683 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995641    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c746bf1f-fe37-447b-a830-317e67295a87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c746bf1f-fe37-447b-a830-317e67295a87" (UID: "c746bf1f-fe37-447b-a830-317e67295a87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:20:46.995765 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995570    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c746bf1f-fe37-447b-a830-317e67295a87" (UID: "c746bf1f-fe37-447b-a830-317e67295a87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:20:46.995765 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.995738    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c746bf1f-fe37-447b-a830-317e67295a87" (UID: "c746bf1f-fe37-447b-a830-317e67295a87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:20:46.997558 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.997535    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c746bf1f-fe37-447b-a830-317e67295a87-kube-api-access-h6cz2" (OuterVolumeSpecName: "kube-api-access-h6cz2") pod "c746bf1f-fe37-447b-a830-317e67295a87" (UID: "c746bf1f-fe37-447b-a830-317e67295a87"). InnerVolumeSpecName "kube-api-access-h6cz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:20:46.997644 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:46.997612    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c746bf1f-fe37-447b-a830-317e67295a87" (UID: "c746bf1f-fe37-447b-a830-317e67295a87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:20:47.095724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.095694    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:20:47.095724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.095719    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c746bf1f-fe37-447b-a830-317e67295a87-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:20:47.095724 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.095730    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c746bf1f-fe37-447b-a830-317e67295a87-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:20:47.095923 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.095739    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6cz2\" (UniqueName: \"kubernetes.io/projected/c746bf1f-fe37-447b-a830-317e67295a87-kube-api-access-h6cz2\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:20:47.095923 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.095748    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c746bf1f-fe37-447b-a830-317e67295a87-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:20:47.239307 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.239277    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"]
Jul 24 22:20:47.241600 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.241575    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-f9j8c"]
Jul 24 22:20:47.326855 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.326830    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c746bf1f-fe37-447b-a830-317e67295a87" path="/var/lib/kubelet/pods/c746bf1f-fe37-447b-a830-317e67295a87/volumes"
Jul 24 22:20:47.553678 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.553562    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:20:47.616888 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.616862    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"]
Jul 24 22:20:47.617130 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617119    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container"
Jul 24 22:20:47.617173 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617132    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container"
Jul 24 22:20:47.617173 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617141    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="storage-initializer"
Jul 24 22:20:47.617173 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617147    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="storage-initializer"
Jul 24 22:20:47.617173 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617159    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kube-rbac-proxy"
Jul 24 22:20:47.617173 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617164    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kube-rbac-proxy"
Jul 24 22:20:47.617323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617205    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kserve-container"
Jul 24 22:20:47.617323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.617215    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c746bf1f-fe37-447b-a830-317e67295a87" containerName="kube-rbac-proxy"
Jul 24 22:20:47.621574 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.621554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.624410 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.624383    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 24 22:20:47.624525 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.624478    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:20:47.624525 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.624485    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:20:47.624762 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.624646    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 24 22:20:47.624762 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.624753    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:20:47.625065 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.625032    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:20:47.625925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.625908    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:20:47.631932 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.631911    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"]
Jul 24 22:20:47.701250 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.701219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zcx\" (UniqueName: \"kubernetes.io/projected/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-api-access-56zcx\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.701389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.701258    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.701389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.701286    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.701389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.701338    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0820134-59f6-4304-9ab1-e9de2c3b3e30-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.701535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.701415    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.794632 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.794600    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:20:47.802727 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.802701    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.802846 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.802736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.802846 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.802761    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0820134-59f6-4304-9ab1-e9de2c3b3e30-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.802846 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.802807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.803002 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.802858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zcx\" (UniqueName: \"kubernetes.io/projected/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-api-access-56zcx\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.803157 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.803134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.803342 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.803326    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.803476 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.803459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.805349 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.805299    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0820134-59f6-4304-9ab1-e9de2c3b3e30-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.811446 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.811425    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zcx\" (UniqueName: \"kubernetes.io/projected/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-api-access-56zcx\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:47.933105 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:47.933079    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:48.062796 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:48.062772    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"]
Jul 24 22:20:48.064840 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:20:48.064812    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0820134_59f6_4304_9ab1_e9de2c3b3e30.slice/crio-4e160f2357a82f5309ec3bac8a8d6eb0c77a564ae448b9f1597d879bbab0a246 WatchSource:0}: Error finding container 4e160f2357a82f5309ec3bac8a8d6eb0c77a564ae448b9f1597d879bbab0a246: Status 404 returned error can't find the container with id 4e160f2357a82f5309ec3bac8a8d6eb0c77a564ae448b9f1597d879bbab0a246
Jul 24 22:20:48.925786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:48.925750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerStarted","Data":"a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a"}
Jul 24 22:20:48.925938 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:48.925807    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerStarted","Data":"4e160f2357a82f5309ec3bac8a8d6eb0c77a564ae448b9f1597d879bbab0a246"}
Jul 24 22:20:52.942064 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:52.942029    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerID="a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a" exitCode=0
Jul 24 22:20:52.942501 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:52.942091    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerDied","Data":"a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a"}
Jul 24 22:20:53.947759 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:53.947709    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerStarted","Data":"07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc"}
Jul 24 22:20:53.948177 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:53.947785    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerStarted","Data":"0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e"}
Jul 24 22:20:53.948177 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:53.948057    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:53.948177 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:53.948113    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:20:53.968949 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:53.968899    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podStartSLOduration=2.972706095 podStartE2EDuration="6.968885206s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.996179111s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:20:53.967213075 +0000 UTC m=+3935.160391039" watchObservedRunningTime="2026-07-24 22:20:53.968885206 +0000 UTC m=+3935.162063139"
Jul 24 22:20:59.956188 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:20:59.956157    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:21:29.957344 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:21:29.957302    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 24 22:21:39.957289 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:21:39.957245    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 24 22:21:49.957680 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:21:49.957613    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 24 22:21:59.957467 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:21:59.957427    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 24 22:22:09.984359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:09.984328    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:22:17.750368 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:17.750334    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"]
Jul 24 22:22:17.750935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:17.750642    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container" containerID="cri-o://0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e" gracePeriod=30
Jul 24 22:22:17.750935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:17.750692    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kube-rbac-proxy" containerID="cri-o://07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc" gracePeriod=30
Jul 24 22:22:17.760394 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:17.760362    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:22:18.264857 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:18.264826    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerID="07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc" exitCode=2
Jul 24 22:22:18.265022 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:18.264880    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerDied","Data":"07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc"}
Jul 24 22:22:19.951761 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:19.951715    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 24 22:22:19.956887 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:19.956861    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 24 22:22:22.855056 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:22.854977    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:22:23.082046 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.082020    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:22:23.093618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.093599    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:22:23.278397 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278374    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-cabundle-cert\") pod \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") "
Jul 24 22:22:23.278774 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278416    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-rbac-proxy-sar-config\") pod \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") "
Jul 24 22:22:23.278774 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278443    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56zcx\" (UniqueName: \"kubernetes.io/projected/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-api-access-56zcx\") pod \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") "
Jul 24 22:22:23.278774 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278488    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kserve-provision-location\") pod \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") "
Jul 24 22:22:23.278774 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278530    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0820134-59f6-4304-9ab1-e9de2c3b3e30-proxy-tls\") pod \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\" (UID: \"d0820134-59f6-4304-9ab1-e9de2c3b3e30\") "
Jul 24 22:22:23.278991 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278813    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0820134-59f6-4304-9ab1-e9de2c3b3e30" (UID: "d0820134-59f6-4304-9ab1-e9de2c3b3e30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:22:23.278991 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278817    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0820134-59f6-4304-9ab1-e9de2c3b3e30" (UID: "d0820134-59f6-4304-9ab1-e9de2c3b3e30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:22:23.278991 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.278892    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0820134-59f6-4304-9ab1-e9de2c3b3e30" (UID: "d0820134-59f6-4304-9ab1-e9de2c3b3e30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:22:23.280734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.280708    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0820134-59f6-4304-9ab1-e9de2c3b3e30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0820134-59f6-4304-9ab1-e9de2c3b3e30" (UID: "d0820134-59f6-4304-9ab1-e9de2c3b3e30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:22:23.281059 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.280990    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-api-access-56zcx" (OuterVolumeSpecName: "kube-api-access-56zcx") pod "d0820134-59f6-4304-9ab1-e9de2c3b3e30" (UID: "d0820134-59f6-4304-9ab1-e9de2c3b3e30"). InnerVolumeSpecName "kube-api-access-56zcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:22:23.283974 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.283951    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerID="0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e" exitCode=0
Jul 24 22:22:23.284075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.283993    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerDied","Data":"0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e"}
Jul 24 22:22:23.284075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.284008    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"
Jul 24 22:22:23.284075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.284039    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq" event={"ID":"d0820134-59f6-4304-9ab1-e9de2c3b3e30","Type":"ContainerDied","Data":"4e160f2357a82f5309ec3bac8a8d6eb0c77a564ae448b9f1597d879bbab0a246"}
Jul 24 22:22:23.284075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.284066    2585 scope.go:117] "RemoveContainer" containerID="07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc"
Jul 24 22:22:23.294784 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.294767    2585 scope.go:117] "RemoveContainer" containerID="0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e"
Jul 24 22:22:23.301600 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.301584    2585 scope.go:117] "RemoveContainer" containerID="a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a"
Jul 24 22:22:23.306540 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.306518    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"]
Jul 24 22:22:23.309210 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.309194    2585 scope.go:117] "RemoveContainer" containerID="07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc"
Jul 24 22:22:23.309473 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:22:23.309448    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc\": container with ID starting with 07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc not found: ID does not exist" containerID="07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc"
Jul 24 22:22:23.309536 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.309480    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc"} err="failed to get container status \"07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc\": rpc error: code = NotFound desc = could not find container \"07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc\": container with ID starting with 07b74ec169a791299baaf46e1a8a83782e64357de3bb730c96b986b8c0f4c2fc not found: ID does not exist"
Jul 24 22:22:23.309536 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.309498    2585 scope.go:117] "RemoveContainer" containerID="0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e"
Jul 24 22:22:23.309790 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:22:23.309774    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e\": container with ID starting with 0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e not found: ID does not exist" containerID="0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e"
Jul 24 22:22:23.309880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.309792    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e"} err="failed to get container status \"0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e\": rpc error: code = NotFound desc = could not find container \"0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e\": container with ID starting with 0324c49f8f580f60173ac4f2b6ee24ea419fadf20e36d97f9a3c352b42aede6e not found: ID does not exist"
Jul 24 22:22:23.309880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.309804    2585 scope.go:117] "RemoveContainer" containerID="a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a"
Jul 24 22:22:23.310026 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:22:23.310010    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a\": container with ID starting with a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a not found: ID does not exist" containerID="a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a"
Jul 24 22:22:23.310069 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.310030    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a"} err="failed to get container status \"a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a\": rpc error: code = NotFound desc = could not find container \"a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a\": container with ID starting with a38ed6c48515cc97e43d0f67a9518cd6333efc5edf1d148dade7b550f05e0e3a not found: ID does not exist"
Jul 24 22:22:23.310311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.310295    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-72qbq"]
Jul 24 22:22:23.327385 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.327362    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" path="/var/lib/kubelet/pods/d0820134-59f6-4304-9ab1-e9de2c3b3e30/volumes"
Jul 24 22:22:23.379360 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.379335    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:22:23.379454 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.379360    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zcx\" (UniqueName: \"kubernetes.io/projected/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kube-api-access-56zcx\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:22:23.379454 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.379376    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0820134-59f6-4304-9ab1-e9de2c3b3e30-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:22:23.379454 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.379388    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0820134-59f6-4304-9ab1-e9de2c3b3e30-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:22:23.379454 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:23.379401    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0820134-59f6-4304-9ab1-e9de2c3b3e30-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:22:24.948297 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:24.948263    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:22:30.050851 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.050813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:22:30.095438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095405    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"]
Jul 24 22:22:30.095731 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095717    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kube-rbac-proxy"
Jul 24 22:22:30.095786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095732    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kube-rbac-proxy"
Jul 24 22:22:30.095786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095746    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="storage-initializer"
Jul 24 22:22:30.095786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095757    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="storage-initializer"
Jul 24 22:22:30.095786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095768    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container"
Jul 24 22:22:30.095786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095773    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container"
Jul 24 22:22:30.095938 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095815    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kserve-container"
Jul 24 22:22:30.095938 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.095825    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0820134-59f6-4304-9ab1-e9de2c3b3e30" containerName="kube-rbac-proxy"
Jul 24 22:22:30.100490 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.100469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.103200 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.103180    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:22:30.103296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.103274    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 24 22:22:30.103296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.103288    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:22:30.104392 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.104367    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:22:30.104392 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.104380    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:22:30.104543 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.104412    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 24 22:22:30.104637 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.104619    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:22:30.113337 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.113312    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"]
Jul 24 22:22:30.131507 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.131461    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9lw\" (UniqueName: \"kubernetes.io/projected/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-api-access-sk9lw\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.131507 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.131504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.131722 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.131561    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-cabundle-cert\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.131722 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.131583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e823d3-a943-4883-9e8f-25f11ef8798c-kserve-provision-location\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.131722 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.131616    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.231935 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.231908    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-cabundle-cert\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232091 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.231944    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e823d3-a943-4883-9e8f-25f11ef8798c-kserve-provision-location\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232091 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.231962    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232091 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.232014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9lw\" (UniqueName: \"kubernetes.io/projected/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-api-access-sk9lw\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232091 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.232032    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232339 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:22:30.232141    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Jul 24 22:22:30.232339 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:22:30.232197    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls podName:a9e823d3-a943-4883-9e8f-25f11ef8798c nodeName:}" failed. No retries permitted until 2026-07-24 22:22:30.732179153 +0000 UTC m=+4031.925357087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls") pod "isvc-sklearn-predictor-6c799687d7-2qq4t" (UID: "a9e823d3-a943-4883-9e8f-25f11ef8798c") : secret "isvc-sklearn-predictor-serving-cert" not found
Jul 24 22:22:30.232441 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.232361    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e823d3-a943-4883-9e8f-25f11ef8798c-kserve-provision-location\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232706 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.232685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.232761 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.232742    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-cabundle-cert\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.241106 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.241082    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9lw\" (UniqueName: \"kubernetes.io/projected/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-api-access-sk9lw\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.604697 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.604644    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:22:30.735413 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.735382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:30.737895 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:30.737878    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls\") pod \"isvc-sklearn-predictor-6c799687d7-2qq4t\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:31.012274 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:31.012224    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:31.140517 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:31.140496    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"]
Jul 24 22:22:31.143004 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:22:31.142980    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e823d3_a943_4883_9e8f_25f11ef8798c.slice/crio-58acaf5aca65e4a068c9d2ab3c3dbe9b12efd56f4ac974905bf1e4f348ee5c6e WatchSource:0}: Error finding container 58acaf5aca65e4a068c9d2ab3c3dbe9b12efd56f4ac974905bf1e4f348ee5c6e: Status 404 returned error can't find the container with id 58acaf5aca65e4a068c9d2ab3c3dbe9b12efd56f4ac974905bf1e4f348ee5c6e
Jul 24 22:22:31.316939 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:31.316860    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerStarted","Data":"075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d"}
Jul 24 22:22:31.316939 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:31.316918    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerStarted","Data":"58acaf5aca65e4a068c9d2ab3c3dbe9b12efd56f4ac974905bf1e4f348ee5c6e"}
Jul 24 22:22:35.335696 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:35.335646    2585 generic.go:356] "Generic (PLEG): container finished" podID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerID="075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d" exitCode=0
Jul 24 22:22:35.335696 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:35.335692    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerDied","Data":"075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d"}
Jul 24 22:22:36.341365 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:36.341332    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerStarted","Data":"8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42"}
Jul 24 22:22:36.341779 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:36.341412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerStarted","Data":"481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88"}
Jul 24 22:22:36.341779 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:36.341695    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:36.362621 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:36.362565    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podStartSLOduration=2.867083696 podStartE2EDuration="6.362547651s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.495463955s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:22:36.360739838 +0000 UTC m=+4037.553917777" watchObservedRunningTime="2026-07-24 22:22:36.362547651 +0000 UTC m=+4037.555725592"
Jul 24 22:22:37.345183 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:37.345150    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:37.346477 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:37.346447    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:22:38.348520 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:38.348483    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:22:43.353698 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:43.353611    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:22:43.354156 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:43.354083    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:22:53.354148 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:22:53.354105    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:23:03.354071 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:23:03.354030    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:23:13.354038 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:23:13.354002    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:23:23.354461 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:23:23.354423    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:23:33.354370 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:23:33.354331    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:23:43.354322 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:23:43.354285    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:23:53.355453 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:23:53.355424    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:24:00.193162 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.193120    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:24:00.265656 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.265614    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"]
Jul 24 22:24:00.266014 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.265969    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" containerID="cri-o://481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88" gracePeriod=30
Jul 24 22:24:00.266014 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.265976    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kube-rbac-proxy" containerID="cri-o://8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42" gracePeriod=30
Jul 24 22:24:00.273799 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.273771    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:24:00.649381 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.649345    2585 generic.go:356] "Generic (PLEG): container finished" podID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerID="8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42" exitCode=2
Jul 24 22:24:00.649536 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:00.649415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerDied","Data":"8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42"}
Jul 24 22:24:03.349730 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:03.349687    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 24 22:24:03.354052 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:03.354029    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 24 22:24:04.617618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.617593    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:24:04.664934 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.664857    2585 generic.go:356] "Generic (PLEG): container finished" podID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerID="481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88" exitCode=0
Jul 24 22:24:04.664934 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.664909    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"
Jul 24 22:24:04.665104 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.664939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerDied","Data":"481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88"}
Jul 24 22:24:04.665104 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.665027    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t" event={"ID":"a9e823d3-a943-4883-9e8f-25f11ef8798c","Type":"ContainerDied","Data":"58acaf5aca65e4a068c9d2ab3c3dbe9b12efd56f4ac974905bf1e4f348ee5c6e"}
Jul 24 22:24:04.665104 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.665053    2585 scope.go:117] "RemoveContainer" containerID="8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42"
Jul 24 22:24:04.672840 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.672825    2585 scope.go:117] "RemoveContainer" containerID="481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88"
Jul 24 22:24:04.679794 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.679767    2585 scope.go:117] "RemoveContainer" containerID="075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d"
Jul 24 22:24:04.686762 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.686746    2585 scope.go:117] "RemoveContainer" containerID="8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42"
Jul 24 22:24:04.686998 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:24:04.686981    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42\": container with ID starting with 8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42 not found: ID does not exist" containerID="8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42"
Jul 24 22:24:04.687046 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.687007    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42"} err="failed to get container status \"8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42\": rpc error: code = NotFound desc = could not find container \"8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42\": container with ID starting with 8a07ac3ce31137e705d274db846ac8dccb0e66319890bf459993221dae9dde42 not found: ID does not exist"
Jul 24 22:24:04.687046 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.687025    2585 scope.go:117] "RemoveContainer" containerID="481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88"
Jul 24 22:24:04.687292 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:24:04.687264    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88\": container with ID starting with 481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88 not found: ID does not exist" containerID="481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88"
Jul 24 22:24:04.687349 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.687290    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88"} err="failed to get container status \"481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88\": rpc error: code = NotFound desc = could not find container \"481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88\": container with ID starting with 481a5a36492527441fb356f041493b6150365747c8bfe91a11a42e050105bf88 not found: ID does not exist"
Jul 24 22:24:04.687349 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.687306    2585 scope.go:117] "RemoveContainer" containerID="075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d"
Jul 24 22:24:04.687533 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:24:04.687517    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d\": container with ID starting with 075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d not found: ID does not exist" containerID="075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d"
Jul 24 22:24:04.687577 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.687536    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d"} err="failed to get container status \"075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d\": rpc error: code = NotFound desc = could not find container \"075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d\": container with ID starting with 075858620fcf51e8e121964d2249000dd769758a62343c24d8376c996ee2393d not found: ID does not exist"
Jul 24 22:24:04.764646 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.764629    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-cabundle-cert\") pod \"a9e823d3-a943-4883-9e8f-25f11ef8798c\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") "
Jul 24 22:24:04.764775 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.764659    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk9lw\" (UniqueName: \"kubernetes.io/projected/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-api-access-sk9lw\") pod \"a9e823d3-a943-4883-9e8f-25f11ef8798c\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") "
Jul 24 22:24:04.764775 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.764711    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-rbac-proxy-sar-config\") pod \"a9e823d3-a943-4883-9e8f-25f11ef8798c\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") "
Jul 24 22:24:04.764896 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.764833    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e823d3-a943-4883-9e8f-25f11ef8798c-kserve-provision-location\") pod \"a9e823d3-a943-4883-9e8f-25f11ef8798c\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") "
Jul 24 22:24:04.764896 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.764886    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls\") pod \"a9e823d3-a943-4883-9e8f-25f11ef8798c\" (UID: \"a9e823d3-a943-4883-9e8f-25f11ef8798c\") "
Jul 24 22:24:04.765128 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.765097    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9e823d3-a943-4883-9e8f-25f11ef8798c" (UID: "a9e823d3-a943-4883-9e8f-25f11ef8798c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:24:04.765128 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.765107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e823d3-a943-4883-9e8f-25f11ef8798c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9e823d3-a943-4883-9e8f-25f11ef8798c" (UID: "a9e823d3-a943-4883-9e8f-25f11ef8798c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:24:04.765128 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.765117    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9e823d3-a943-4883-9e8f-25f11ef8798c" (UID: "a9e823d3-a943-4883-9e8f-25f11ef8798c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:24:04.766916 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.766887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-api-access-sk9lw" (OuterVolumeSpecName: "kube-api-access-sk9lw") pod "a9e823d3-a943-4883-9e8f-25f11ef8798c" (UID: "a9e823d3-a943-4883-9e8f-25f11ef8798c"). InnerVolumeSpecName "kube-api-access-sk9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:24:04.766916 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.766901    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9e823d3-a943-4883-9e8f-25f11ef8798c" (UID: "a9e823d3-a943-4883-9e8f-25f11ef8798c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:24:04.866322 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.866300    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9e823d3-a943-4883-9e8f-25f11ef8798c-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:24:04.866322 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.866320    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:24:04.866474 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.866329    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9lw\" (UniqueName: \"kubernetes.io/projected/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-api-access-sk9lw\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:24:04.866474 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.866340    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9e823d3-a943-4883-9e8f-25f11ef8798c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:24:04.866474 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.866348    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9e823d3-a943-4883-9e8f-25f11ef8798c-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:24:04.986913 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.986887    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"]
Jul 24 22:24:04.990793 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:04.990772    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6c799687d7-2qq4t"]
Jul 24 22:24:05.326885 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.326807    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" path="/var/lib/kubelet/pods/a9e823d3-a943-4883-9e8f-25f11ef8798c/volumes"
Jul 24 22:24:05.356860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.356831    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:24:05.412740 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.412710    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"]
Jul 24 22:24:05.413050 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413031    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="storage-initializer"
Jul 24 22:24:05.413138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413084    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="storage-initializer"
Jul 24 22:24:05.413138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413097    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container"
Jul 24 22:24:05.413138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413106    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container"
Jul 24 22:24:05.413138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413126    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kube-rbac-proxy"
Jul 24 22:24:05.413138 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413134    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kube-rbac-proxy"
Jul 24 22:24:05.413393 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413222    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kserve-container"
Jul 24 22:24:05.413393 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.413235    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e823d3-a943-4883-9e8f-25f11ef8798c" containerName="kube-rbac-proxy"
Jul 24 22:24:05.417828 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.417809    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.420462 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420436    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 24 22:24:05.420776 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420759    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:24:05.420858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420774    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:24:05.420858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420819    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:24:05.420858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420834    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:24:05.421019 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420774    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 24 22:24:05.421019 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.420881    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:24:05.426520 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.426499    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"]
Jul 24 22:24:05.573149 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.573113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.573321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.573161    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.573321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.573182    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt5m\" (UniqueName: \"kubernetes.io/projected/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-api-access-ggt5m\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.573321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.573239    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cab9250-02bd-482b-8010-9d8b5c3759b5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.573321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.573258    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cab9250-02bd-482b-8010-9d8b5c3759b5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.673641 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.673607    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.673641 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.673639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt5m\" (UniqueName: \"kubernetes.io/projected/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-api-access-ggt5m\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.674078 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.673693    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cab9250-02bd-482b-8010-9d8b5c3759b5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.674078 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.673712    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cab9250-02bd-482b-8010-9d8b5c3759b5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.674078 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.673745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.674199 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.674141    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cab9250-02bd-482b-8010-9d8b5c3759b5-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.674310 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.674292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.674378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.674362    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.676419 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.676395    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cab9250-02bd-482b-8010-9d8b5c3759b5-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.682294 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.682273    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt5m\" (UniqueName: \"kubernetes.io/projected/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-api-access-ggt5m\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.690702 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.690655    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:24:05.728785 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.728756    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:05.856157 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:05.855984    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"]
Jul 24 22:24:05.858778 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:24:05.858756    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cab9250_02bd_482b_8010_9d8b5c3759b5.slice/crio-aa2f639ddf91a9b479ffd0b690ea42fb3299d629f8be8741004d1f1d0fd46393 WatchSource:0}: Error finding container aa2f639ddf91a9b479ffd0b690ea42fb3299d629f8be8741004d1f1d0fd46393: Status 404 returned error can't find the container with id aa2f639ddf91a9b479ffd0b690ea42fb3299d629f8be8741004d1f1d0fd46393
Jul 24 22:24:06.674929 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:06.674890    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerStarted","Data":"a1902f8593caeed71bb2580d08d48746f3061c1b76dad8b5195b908600d7d215"}
Jul 24 22:24:06.675398 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:06.674968    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerStarted","Data":"aa2f639ddf91a9b479ffd0b690ea42fb3299d629f8be8741004d1f1d0fd46393"}
Jul 24 22:24:09.687870 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:09.687836    2585 generic.go:356] "Generic (PLEG): container finished" podID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerID="a1902f8593caeed71bb2580d08d48746f3061c1b76dad8b5195b908600d7d215" exitCode=0
Jul 24 22:24:09.688196 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:09.687911    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerDied","Data":"a1902f8593caeed71bb2580d08d48746f3061c1b76dad8b5195b908600d7d215"}
Jul 24 22:24:10.693899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:10.693862    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerStarted","Data":"ff5afb0659402315d0327cc4101697421ec37d76ec2818a151b067e76a0514f1"}
Jul 24 22:24:10.694304 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:10.693932    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerStarted","Data":"842b4597278c9ab0b9cc2a2a6791c72af968f280829d83e1886eeb81c20fdb14"}
Jul 24 22:24:10.694304 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:10.694224    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:10.694304 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:10.694270    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:10.716594 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:10.716545    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podStartSLOduration=2.00788162 podStartE2EDuration="5.716530801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.708649181s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:24:10.714797924 +0000 UTC m=+4131.907975864" watchObservedRunningTime="2026-07-24 22:24:10.716530801 +0000 UTC m=+4131.909708738"
Jul 24 22:24:16.702638 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:16.702610    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:24:46.780598 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:46.780549    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 24 22:24:56.706127 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:24:56.706094    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:25:05.497227 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:05.497192    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"]
Jul 24 22:25:05.497635 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:05.497506    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kserve-container" containerID="cri-o://842b4597278c9ab0b9cc2a2a6791c72af968f280829d83e1886eeb81c20fdb14" gracePeriod=30
Jul 24 22:25:05.497635 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:05.497570    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kube-rbac-proxy" containerID="cri-o://ff5afb0659402315d0327cc4101697421ec37d76ec2818a151b067e76a0514f1" gracePeriod=30
Jul 24 22:25:05.522534 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:05.522510    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:25:05.900068 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:05.900040    2585 generic.go:356] "Generic (PLEG): container finished" podID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerID="ff5afb0659402315d0327cc4101697421ec37d76ec2818a151b067e76a0514f1" exitCode=2
Jul 24 22:25:05.900231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:05.900115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerDied","Data":"ff5afb0659402315d0327cc4101697421ec37d76ec2818a151b067e76a0514f1"}
Jul 24 22:25:06.697961 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:06.697918    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 24 22:25:07.744823 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:07.744781    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 22:25:10.633991 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:10.633938    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:25:10.746810 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:10.746771    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:25:11.698171 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:11.698111    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 24 22:25:11.925521 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:11.925489    2585 generic.go:356] "Generic (PLEG): container finished" podID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerID="842b4597278c9ab0b9cc2a2a6791c72af968f280829d83e1886eeb81c20fdb14" exitCode=0
Jul 24 22:25:11.925633 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:11.925544    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerDied","Data":"842b4597278c9ab0b9cc2a2a6791c72af968f280829d83e1886eeb81c20fdb14"}
Jul 24 22:25:12.046017 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.045996    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:25:12.175880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.175851    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cab9250-02bd-482b-8010-9d8b5c3759b5-proxy-tls\") pod \"2cab9250-02bd-482b-8010-9d8b5c3759b5\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") "
Jul 24 22:25:12.176040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.175889    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cab9250-02bd-482b-8010-9d8b5c3759b5-kserve-provision-location\") pod \"2cab9250-02bd-482b-8010-9d8b5c3759b5\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") "
Jul 24 22:25:12.176040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.175948    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-cabundle-cert\") pod \"2cab9250-02bd-482b-8010-9d8b5c3759b5\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") "
Jul 24 22:25:12.176040 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.176007    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggt5m\" (UniqueName: \"kubernetes.io/projected/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-api-access-ggt5m\") pod \"2cab9250-02bd-482b-8010-9d8b5c3759b5\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") "
Jul 24 22:25:12.176149 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.176052    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-rbac-proxy-sar-config\") pod \"2cab9250-02bd-482b-8010-9d8b5c3759b5\" (UID: \"2cab9250-02bd-482b-8010-9d8b5c3759b5\") "
Jul 24 22:25:12.176400 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.176313    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cab9250-02bd-482b-8010-9d8b5c3759b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2cab9250-02bd-482b-8010-9d8b5c3759b5" (UID: "2cab9250-02bd-482b-8010-9d8b5c3759b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:25:12.176515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.176404    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2cab9250-02bd-482b-8010-9d8b5c3759b5" (UID: "2cab9250-02bd-482b-8010-9d8b5c3759b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:25:12.176515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.176499    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2cab9250-02bd-482b-8010-9d8b5c3759b5" (UID: "2cab9250-02bd-482b-8010-9d8b5c3759b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:25:12.178219 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.178196    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cab9250-02bd-482b-8010-9d8b5c3759b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2cab9250-02bd-482b-8010-9d8b5c3759b5" (UID: "2cab9250-02bd-482b-8010-9d8b5c3759b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:25:12.178320 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.178298    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-api-access-ggt5m" (OuterVolumeSpecName: "kube-api-access-ggt5m") pod "2cab9250-02bd-482b-8010-9d8b5c3759b5" (UID: "2cab9250-02bd-482b-8010-9d8b5c3759b5"). InnerVolumeSpecName "kube-api-access-ggt5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:25:12.276628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.276561    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt5m\" (UniqueName: \"kubernetes.io/projected/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-api-access-ggt5m\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:25:12.276628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.276581    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:25:12.276628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.276591    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cab9250-02bd-482b-8010-9d8b5c3759b5-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:25:12.276628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.276610    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cab9250-02bd-482b-8010-9d8b5c3759b5-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:25:12.276628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.276618    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cab9250-02bd-482b-8010-9d8b5c3759b5-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:25:12.931813 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.931780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4" event={"ID":"2cab9250-02bd-482b-8010-9d8b5c3759b5","Type":"ContainerDied","Data":"aa2f639ddf91a9b479ffd0b690ea42fb3299d629f8be8741004d1f1d0fd46393"}
Jul 24 22:25:12.931813 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.931806    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"
Jul 24 22:25:12.932312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.931849    2585 scope.go:117] "RemoveContainer" containerID="ff5afb0659402315d0327cc4101697421ec37d76ec2818a151b067e76a0514f1"
Jul 24 22:25:12.940514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.940494    2585 scope.go:117] "RemoveContainer" containerID="842b4597278c9ab0b9cc2a2a6791c72af968f280829d83e1886eeb81c20fdb14"
Jul 24 22:25:12.947930 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.947913    2585 scope.go:117] "RemoveContainer" containerID="a1902f8593caeed71bb2580d08d48746f3061c1b76dad8b5195b908600d7d215"
Jul 24 22:25:12.953963 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.953928    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"]
Jul 24 22:25:12.959575 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:12.959554    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6w7m4"]
Jul 24 22:25:13.327094 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:13.327002    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" path="/var/lib/kubelet/pods/2cab9250-02bd-482b-8010-9d8b5c3759b5/volumes"
Jul 24 22:25:42.663835 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:42.663721    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:25:47.758845 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.758802    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:25:47.837517 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837487    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"]
Jul 24 22:25:47.837822 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837808    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kserve-container"
Jul 24 22:25:47.837875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837825    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kserve-container"
Jul 24 22:25:47.837875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837838    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="storage-initializer"
Jul 24 22:25:47.837875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837843    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="storage-initializer"
Jul 24 22:25:47.837875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837852    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kube-rbac-proxy"
Jul 24 22:25:47.837875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837857    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kube-rbac-proxy"
Jul 24 22:25:47.838034 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837914    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kserve-container"
Jul 24 22:25:47.838034 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.837921    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cab9250-02bd-482b-8010-9d8b5c3759b5" containerName="kube-rbac-proxy"
Jul 24 22:25:47.840916 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.840901    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:47.845302 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.845274    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:25:47.845302 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.845291    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:25:47.845494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.845313    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:25:47.845494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.845313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:25:47.845494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.845463    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:25:47.846377 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.846363    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:25:47.846452 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.846366    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 24 22:25:47.863588 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.863560    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"]
Jul 24 22:25:47.920146 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.920121    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6807afd0-e5b0-420e-ba0b-f994d001df5f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:47.920340 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.920159    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6807afd0-e5b0-420e-ba0b-f994d001df5f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:47.920340 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.920178    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:47.920340 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.920231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpfq\" (UniqueName: \"kubernetes.io/projected/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-api-access-jvpfq\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:47.920340 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:47.920302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.020961    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6807afd0-e5b0-420e-ba0b-f994d001df5f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.020996    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6807afd0-e5b0-420e-ba0b-f994d001df5f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.021014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021334 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.021032    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpfq\" (UniqueName: \"kubernetes.io/projected/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-api-access-jvpfq\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021334 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.021052    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021458 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.021439    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6807afd0-e5b0-420e-ba0b-f994d001df5f-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021688 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.021641    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.021797 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.021778    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.023728 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.023707    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6807afd0-e5b0-420e-ba0b-f994d001df5f-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.029742 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.029724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpfq\" (UniqueName: \"kubernetes.io/projected/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-api-access-jvpfq\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.153068 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.153041    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:48.282104 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.282079    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"]
Jul 24 22:25:48.284624 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:25:48.284586    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6807afd0_e5b0_420e_ba0b_f994d001df5f.slice/crio-e6acea29d2e728a0bd483ff9181cac0aa370d12cc68c62fb2c2ebe07173ee4aa WatchSource:0}: Error finding container e6acea29d2e728a0bd483ff9181cac0aa370d12cc68c62fb2c2ebe07173ee4aa: Status 404 returned error can't find the container with id e6acea29d2e728a0bd483ff9181cac0aa370d12cc68c62fb2c2ebe07173ee4aa
Jul 24 22:25:48.917947 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:48.917910    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:25:49.058649 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:49.058614    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerStarted","Data":"275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b"}
Jul 24 22:25:49.058865 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:49.058691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerStarted","Data":"e6acea29d2e728a0bd483ff9181cac0aa370d12cc68c62fb2c2ebe07173ee4aa"}
Jul 24 22:25:53.072683 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:53.072642    2585 generic.go:356] "Generic (PLEG): container finished" podID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerID="275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b" exitCode=0
Jul 24 22:25:53.073170 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:53.072716    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerDied","Data":"275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b"}
Jul 24 22:25:54.078337 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:54.078301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerStarted","Data":"df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824"}
Jul 24 22:25:54.078737 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:54.078363    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerStarted","Data":"af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe"}
Jul 24 22:25:54.078737 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:54.078561    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:54.078737 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:54.078618    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:25:54.101638 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:25:54.101585    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" podStartSLOduration=3.35116083 podStartE2EDuration="7.101570285s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750409455s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:25:54.099328124 +0000 UTC m=+4235.292506060" watchObservedRunningTime="2026-07-24 22:25:54.101570285 +0000 UTC m=+4235.294748223"
Jul 24 22:26:00.088266 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:00.088235    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:26:30.181127 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:30.181085    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 24 22:26:40.090872 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:40.090838    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:26:48.019573 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:48.019541    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"]
Jul 24 22:26:48.019984 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:48.019798    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kserve-container" containerID="cri-o://af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe" gracePeriod=30
Jul 24 22:26:48.019984 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:48.019827    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kube-rbac-proxy" containerID="cri-o://df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824" gracePeriod=30
Jul 24 22:26:48.027118 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:48.027094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:26:48.273895 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:48.273825    2585 generic.go:356] "Generic (PLEG): container finished" podID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerID="df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824" exitCode=2
Jul 24 22:26:48.273895 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:48.273886    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerDied","Data":"df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824"}
Jul 24 22:26:50.083075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:50.083034    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 24 22:26:51.131855 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:51.131810    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 22:26:53.032492 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.032452    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:26:53.065907 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.065879    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"]
Jul 24 22:26:53.069282 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.069266    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.071926 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.071904    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 24 22:26:53.072122 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.072109    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 24 22:26:53.080732 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.080708    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"]
Jul 24 22:26:53.186146 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.186100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.186146 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.186149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.186422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.186166    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczfc\" (UniqueName: \"kubernetes.io/projected/72b99ee3-049c-4790-9ae1-643c2358d351-kube-api-access-xczfc\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.186422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.186271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.186422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.186312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72b99ee3-049c-4790-9ae1-643c2358d351-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287198 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287198 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287140    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287198 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287158    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xczfc\" (UniqueName: \"kubernetes.io/projected/72b99ee3-049c-4790-9ae1-643c2358d351-kube-api-access-xczfc\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287198 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287515 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287217    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72b99ee3-049c-4790-9ae1-643c2358d351-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287515 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:26:53.287297    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 24 22:26:53.287515 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:26:53.287371    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls podName:72b99ee3-049c-4790-9ae1-643c2358d351 nodeName:}" failed. No retries permitted until 2026-07-24 22:26:53.78735052 +0000 UTC m=+4294.980528442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls") pod "isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" (UID: "72b99ee3-049c-4790-9ae1-643c2358d351") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 24 22:26:53.287742 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287719    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72b99ee3-049c-4790-9ae1-643c2358d351-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287896 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287880    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.287973 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.287959    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.296609 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.296570    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczfc\" (UniqueName: \"kubernetes.io/projected/72b99ee3-049c-4790-9ae1-643c2358d351-kube-api-access-xczfc\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.571942 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.571849    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:26:53.792016 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.791977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.794597 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.794578    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls\") pod \"isvc-sklearn-v2-predictor-59d6dcff47-8nhk6\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:53.981455 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:53.981416    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:54.108888 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:54.108861    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"]
Jul 24 22:26:54.111283 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:26:54.111252    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b99ee3_049c_4790_9ae1_643c2358d351.slice/crio-7a0b190d813734301912599edbb08a7aaaf85c0d76c1f38fb932a5be1c236e2a WatchSource:0}: Error finding container 7a0b190d813734301912599edbb08a7aaaf85c0d76c1f38fb932a5be1c236e2a: Status 404 returned error can't find the container with id 7a0b190d813734301912599edbb08a7aaaf85c0d76c1f38fb932a5be1c236e2a
Jul 24 22:26:54.302437 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:54.302339    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerStarted","Data":"c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0"}
Jul 24 22:26:54.302437 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:54.302412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerStarted","Data":"7a0b190d813734301912599edbb08a7aaaf85c0d76c1f38fb932a5be1c236e2a"}
Jul 24 22:26:54.956216 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:54.956190    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:26:55.101957 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.101892    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6807afd0-e5b0-420e-ba0b-f994d001df5f-kserve-provision-location\") pod \"6807afd0-e5b0-420e-ba0b-f994d001df5f\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") "
Jul 24 22:26:55.101957 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.101950    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvpfq\" (UniqueName: \"kubernetes.io/projected/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-api-access-jvpfq\") pod \"6807afd0-e5b0-420e-ba0b-f994d001df5f\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") "
Jul 24 22:26:55.102117 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.101966    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-cabundle-cert\") pod \"6807afd0-e5b0-420e-ba0b-f994d001df5f\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") "
Jul 24 22:26:55.102117 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.101993    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6807afd0-e5b0-420e-ba0b-f994d001df5f-proxy-tls\") pod \"6807afd0-e5b0-420e-ba0b-f994d001df5f\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") "
Jul 24 22:26:55.102117 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.102026    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-rbac-proxy-sar-config\") pod \"6807afd0-e5b0-420e-ba0b-f994d001df5f\" (UID: \"6807afd0-e5b0-420e-ba0b-f994d001df5f\") "
Jul 24 22:26:55.102236 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.102203    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6807afd0-e5b0-420e-ba0b-f994d001df5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6807afd0-e5b0-420e-ba0b-f994d001df5f" (UID: "6807afd0-e5b0-420e-ba0b-f994d001df5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:26:55.102390 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.102361    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6807afd0-e5b0-420e-ba0b-f994d001df5f" (UID: "6807afd0-e5b0-420e-ba0b-f994d001df5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:26:55.102504 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.102416    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6807afd0-e5b0-420e-ba0b-f994d001df5f" (UID: "6807afd0-e5b0-420e-ba0b-f994d001df5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:26:55.104233 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.104212    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-api-access-jvpfq" (OuterVolumeSpecName: "kube-api-access-jvpfq") pod "6807afd0-e5b0-420e-ba0b-f994d001df5f" (UID: "6807afd0-e5b0-420e-ba0b-f994d001df5f"). InnerVolumeSpecName "kube-api-access-jvpfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:26:55.104355 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.104236    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6807afd0-e5b0-420e-ba0b-f994d001df5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6807afd0-e5b0-420e-ba0b-f994d001df5f" (UID: "6807afd0-e5b0-420e-ba0b-f994d001df5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:26:55.202892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.202865    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpfq\" (UniqueName: \"kubernetes.io/projected/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-api-access-jvpfq\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:26:55.202892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.202887    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:26:55.202892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.202897    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6807afd0-e5b0-420e-ba0b-f994d001df5f-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:26:55.203266 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.202905    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6807afd0-e5b0-420e-ba0b-f994d001df5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:26:55.203266 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.202912    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6807afd0-e5b0-420e-ba0b-f994d001df5f-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:26:55.307828 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.307798    2585 generic.go:356] "Generic (PLEG): container finished" podID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerID="af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe" exitCode=0
Jul 24 22:26:55.307959 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.307866    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"
Jul 24 22:26:55.307959 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.307906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerDied","Data":"af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe"}
Jul 24 22:26:55.308030 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.307956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6" event={"ID":"6807afd0-e5b0-420e-ba0b-f994d001df5f","Type":"ContainerDied","Data":"e6acea29d2e728a0bd483ff9181cac0aa370d12cc68c62fb2c2ebe07173ee4aa"}
Jul 24 22:26:55.308030 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.307985    2585 scope.go:117] "RemoveContainer" containerID="df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824"
Jul 24 22:26:55.316720 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.316703    2585 scope.go:117] "RemoveContainer" containerID="af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe"
Jul 24 22:26:55.324707 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.324690    2585 scope.go:117] "RemoveContainer" containerID="275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b"
Jul 24 22:26:55.329285 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.329263    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"]
Jul 24 22:26:55.334124 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.334049    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-xzbl6"]
Jul 24 22:26:55.335962 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.335942    2585 scope.go:117] "RemoveContainer" containerID="df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824"
Jul 24 22:26:55.336224 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:26:55.336206    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824\": container with ID starting with df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824 not found: ID does not exist" containerID="df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824"
Jul 24 22:26:55.336272 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.336232    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824"} err="failed to get container status \"df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824\": rpc error: code = NotFound desc = could not find container \"df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824\": container with ID starting with df0eb3f7cadeccc3ac6242434a6b7a6ed35b77268713625a4a2c7591d7641824 not found: ID does not exist"
Jul 24 22:26:55.336272 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.336249    2585 scope.go:117] "RemoveContainer" containerID="af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe"
Jul 24 22:26:55.336503 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:26:55.336486    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe\": container with ID starting with af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe not found: ID does not exist" containerID="af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe"
Jul 24 22:26:55.336554 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.336506    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe"} err="failed to get container status \"af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe\": rpc error: code = NotFound desc = could not find container \"af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe\": container with ID starting with af0e30e38081fc507d34f0e3fb9a1c3b7530645eadd5e31389169e09193c2cfe not found: ID does not exist"
Jul 24 22:26:55.336554 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.336518    2585 scope.go:117] "RemoveContainer" containerID="275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b"
Jul 24 22:26:55.336820 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:26:55.336802    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b\": container with ID starting with 275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b not found: ID does not exist" containerID="275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b"
Jul 24 22:26:55.336865 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:55.336826    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b"} err="failed to get container status \"275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b\": rpc error: code = NotFound desc = could not find container \"275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b\": container with ID starting with 275836b6acffc06867a27508e4970d58da9cfb7ef437483ce22646ff5f4f8e3b not found: ID does not exist"
Jul 24 22:26:57.326529 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:57.326481    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" path="/var/lib/kubelet/pods/6807afd0-e5b0-420e-ba0b-f994d001df5f/volumes"
Jul 24 22:26:58.320622 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:58.320584    2585 generic.go:356] "Generic (PLEG): container finished" podID="72b99ee3-049c-4790-9ae1-643c2358d351" containerID="c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0" exitCode=0
Jul 24 22:26:58.320784 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:58.320678    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerDied","Data":"c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0"}
Jul 24 22:26:59.326635 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:59.326596    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerStarted","Data":"e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6"}
Jul 24 22:26:59.327083 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:59.326655    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerStarted","Data":"5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175"}
Jul 24 22:26:59.327083 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:59.327005    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:59.327205 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:59.327140    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:26:59.328584 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:59.328560    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:26:59.347516 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:26:59.347474    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podStartSLOduration=2.4358884610000002 podStartE2EDuration="6.347460822s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.911572361s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:26:59.345585818 +0000 UTC m=+4300.538763767" watchObservedRunningTime="2026-07-24 22:26:59.347460822 +0000 UTC m=+4300.540638985"
Jul 24 22:27:00.329887 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:00.329854    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:01.333465 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:01.333430    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:06.337745 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:06.337717    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:27:06.338198 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:06.338170    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:16.338922 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:16.338831    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:18.685018 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:18.684972    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:27:26.338819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:26.338774    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:34.866151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:34.866115    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:27:36.339070 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:36.339030    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:46.338580 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:46.338537    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:27:56.338813 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:27:56.338770    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:28:06.338189 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:06.338146    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:28:16.339848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:16.339817    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:28:23.267115 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:23.267080    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"]
Jul 24 22:28:23.267540 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:23.267326    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" containerID="cri-o://5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175" gracePeriod=30
Jul 24 22:28:23.267540 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:23.267357    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kube-rbac-proxy" containerID="cri-o://e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6" gracePeriod=30
Jul 24 22:28:23.289577 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:23.289554    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:28:23.638439 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:23.638407    2585 generic.go:356] "Generic (PLEG): container finished" podID="72b99ee3-049c-4790-9ae1-643c2358d351" containerID="e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6" exitCode=2
Jul 24 22:28:23.638598 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:23.638455    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerDied","Data":"e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6"}
Jul 24 22:28:26.334598 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:26.334557    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 24 22:28:26.338819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:26.338794    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 24 22:28:27.509035 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.509010    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:28:27.653444 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.653411    2585 generic.go:356] "Generic (PLEG): container finished" podID="72b99ee3-049c-4790-9ae1-643c2358d351" containerID="5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175" exitCode=0
Jul 24 22:28:27.653609 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.653482    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"
Jul 24 22:28:27.653609 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.653489    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerDied","Data":"5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175"}
Jul 24 22:28:27.653609 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.653546    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6" event={"ID":"72b99ee3-049c-4790-9ae1-643c2358d351","Type":"ContainerDied","Data":"7a0b190d813734301912599edbb08a7aaaf85c0d76c1f38fb932a5be1c236e2a"}
Jul 24 22:28:27.653609 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.653573    2585 scope.go:117] "RemoveContainer" containerID="e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6"
Jul 24 22:28:27.661048 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.661032    2585 scope.go:117] "RemoveContainer" containerID="5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175"
Jul 24 22:28:27.667980 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.667965    2585 scope.go:117] "RemoveContainer" containerID="c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0"
Jul 24 22:28:27.674801 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.674784    2585 scope.go:117] "RemoveContainer" containerID="e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6"
Jul 24 22:28:27.675043 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:28:27.675025    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6\": container with ID starting with e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6 not found: ID does not exist" containerID="e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6"
Jul 24 22:28:27.675087 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.675052    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6"} err="failed to get container status \"e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6\": rpc error: code = NotFound desc = could not find container \"e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6\": container with ID starting with e5fd35c5901c82fdb472208a9c48757b8a872e132924c6922b6dc7e09c6a36b6 not found: ID does not exist"
Jul 24 22:28:27.675087 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.675067    2585 scope.go:117] "RemoveContainer" containerID="5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175"
Jul 24 22:28:27.675284 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:28:27.675265    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175\": container with ID starting with 5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175 not found: ID does not exist" containerID="5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175"
Jul 24 22:28:27.675320 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.675292    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175"} err="failed to get container status \"5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175\": rpc error: code = NotFound desc = could not find container \"5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175\": container with ID starting with 5d87b7669200e482d5bf86e2ba0715f4ab9ffb0e8e2d92c552a8ed4bb5c25175 not found: ID does not exist"
Jul 24 22:28:27.675320 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.675307    2585 scope.go:117] "RemoveContainer" containerID="c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0"
Jul 24 22:28:27.675501 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:28:27.675485    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0\": container with ID starting with c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0 not found: ID does not exist" containerID="c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0"
Jul 24 22:28:27.675541 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.675505    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0"} err="failed to get container status \"c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0\": rpc error: code = NotFound desc = could not find container \"c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0\": container with ID starting with c568c4447db47769573deda2eccd5078c7f5aa174ca1d732d76d3d6d8176c6c0 not found: ID does not exist"
Jul 24 22:28:27.676743 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.676729    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-cabundle-cert\") pod \"72b99ee3-049c-4790-9ae1-643c2358d351\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") "
Jul 24 22:28:27.676805 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.676768    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls\") pod \"72b99ee3-049c-4790-9ae1-643c2358d351\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") "
Jul 24 22:28:27.676805 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.676792    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-kube-rbac-proxy-sar-config\") pod \"72b99ee3-049c-4790-9ae1-643c2358d351\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") "
Jul 24 22:28:27.676907 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.676812    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xczfc\" (UniqueName: \"kubernetes.io/projected/72b99ee3-049c-4790-9ae1-643c2358d351-kube-api-access-xczfc\") pod \"72b99ee3-049c-4790-9ae1-643c2358d351\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") "
Jul 24 22:28:27.676907 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.676833    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72b99ee3-049c-4790-9ae1-643c2358d351-kserve-provision-location\") pod \"72b99ee3-049c-4790-9ae1-643c2358d351\" (UID: \"72b99ee3-049c-4790-9ae1-643c2358d351\") "
Jul 24 22:28:27.677105 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.677084    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72b99ee3-049c-4790-9ae1-643c2358d351" (UID: "72b99ee3-049c-4790-9ae1-643c2358d351"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:28:27.677171 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.677150    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72b99ee3-049c-4790-9ae1-643c2358d351" (UID: "72b99ee3-049c-4790-9ae1-643c2358d351"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:28:27.677209 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.677178    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b99ee3-049c-4790-9ae1-643c2358d351-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72b99ee3-049c-4790-9ae1-643c2358d351" (UID: "72b99ee3-049c-4790-9ae1-643c2358d351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:28:27.678875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.678850    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72b99ee3-049c-4790-9ae1-643c2358d351" (UID: "72b99ee3-049c-4790-9ae1-643c2358d351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:28:27.678966 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.678898    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b99ee3-049c-4790-9ae1-643c2358d351-kube-api-access-xczfc" (OuterVolumeSpecName: "kube-api-access-xczfc") pod "72b99ee3-049c-4790-9ae1-643c2358d351" (UID: "72b99ee3-049c-4790-9ae1-643c2358d351"). InnerVolumeSpecName "kube-api-access-xczfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:28:27.777707 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.777683    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:28:27.777707 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.777706    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72b99ee3-049c-4790-9ae1-643c2358d351-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:28:27.777877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.777720    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72b99ee3-049c-4790-9ae1-643c2358d351-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:28:27.777877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.777733    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xczfc\" (UniqueName: \"kubernetes.io/projected/72b99ee3-049c-4790-9ae1-643c2358d351-kube-api-access-xczfc\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:28:27.777877 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.777745    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72b99ee3-049c-4790-9ae1-643c2358d351-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:28:27.975977 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.975946    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"]
Jul 24 22:28:27.979565 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:27.979543    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-59d6dcff47-8nhk6"]
Jul 24 22:28:28.362613 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.362540    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:28:28.432045 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432008    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"]
Jul 24 22:28:28.432343 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432326    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432346    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432361    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kserve-container"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432369    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kserve-container"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432389    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kube-rbac-proxy"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432398    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kube-rbac-proxy"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432408    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="storage-initializer"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432416    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="storage-initializer"
Jul 24 22:28:28.432426 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432425    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kube-rbac-proxy"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432433    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kube-rbac-proxy"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432451    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="storage-initializer"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432459    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="storage-initializer"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432521    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kserve-container"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432534    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kserve-container"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432543    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" containerName="kube-rbac-proxy"
Jul 24 22:28:28.432876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.432555    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6807afd0-e5b0-420e-ba0b-f994d001df5f" containerName="kube-rbac-proxy"
Jul 24 22:28:28.440364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.440340    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.442945 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.442922    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 24 22:28:28.443107 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.443084    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:28:28.443191 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.443130    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:28:28.443191 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.443162    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 24 22:28:28.443313 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.443251    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:28:28.443409 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.443383    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:28:28.443513 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.443456    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:28:28.448465 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.448330    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"]
Jul 24 22:28:28.585362 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.585336    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.585734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.585378    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs4q4\" (UniqueName: \"kubernetes.io/projected/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-api-access-gs4q4\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.585734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.585397    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.585734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.585420    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77bc93e9-85c4-4c09-94ba-9444aae93411-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.585734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.585449    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.686554 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.686528    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.686708 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.686570    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4q4\" (UniqueName: \"kubernetes.io/projected/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-api-access-gs4q4\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.686708 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.686588    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.686819 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:28:28.686703    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 24 22:28:28.686819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.686719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77bc93e9-85c4-4c09-94ba-9444aae93411-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.686819 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.686762    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.686819 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:28:28.686778    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls podName:77bc93e9-85c4-4c09-94ba-9444aae93411 nodeName:}" failed. No retries permitted until 2026-07-24 22:28:29.186757909 +0000 UTC m=+4390.379935840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" (UID: "77bc93e9-85c4-4c09-94ba-9444aae93411") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 24 22:28:28.687136 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.687113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77bc93e9-85c4-4c09-94ba-9444aae93411-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.687303 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.687286    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.687383 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.687367    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.697691 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.697657    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4q4\" (UniqueName: \"kubernetes.io/projected/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-api-access-gs4q4\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:28.806990 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:28.806955    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:28:29.190025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.189990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:29.192435 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.192402    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:29.326640 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.326600    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b99ee3-049c-4790-9ae1-643c2358d351" path="/var/lib/kubelet/pods/72b99ee3-049c-4790-9ae1-643c2358d351/volumes"
Jul 24 22:28:29.351938 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.351911    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:29.483516 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.483483    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"]
Jul 24 22:28:29.485584 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:28:29.485551    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bc93e9_85c4_4c09_94ba_9444aae93411.slice/crio-07430f10ff6171179285bef19c1e6bfedff3956be12831efd9ac2f45c81ef2d1 WatchSource:0}: Error finding container 07430f10ff6171179285bef19c1e6bfedff3956be12831efd9ac2f45c81ef2d1: Status 404 returned error can't find the container with id 07430f10ff6171179285bef19c1e6bfedff3956be12831efd9ac2f45c81ef2d1
Jul 24 22:28:29.662817 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.662778    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerStarted","Data":"5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c"}
Jul 24 22:28:29.663157 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:29.662855    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerStarted","Data":"07430f10ff6171179285bef19c1e6bfedff3956be12831efd9ac2f45c81ef2d1"}
Jul 24 22:28:33.680239 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:33.680202    2585 generic.go:356] "Generic (PLEG): container finished" podID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerID="5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c" exitCode=0
Jul 24 22:28:33.680616 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:33.680277    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerDied","Data":"5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c"}
Jul 24 22:28:34.686809 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:34.686777    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerStarted","Data":"8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e"}
Jul 24 22:28:34.687201 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:34.686839    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerStarted","Data":"935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4"}
Jul 24 22:28:34.687201 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:34.687083    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:34.718639 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:34.713254    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podStartSLOduration=2.753614755 podStartE2EDuration="6.713235534s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959620779s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:28:34.712181346 +0000 UTC m=+4395.905359284" watchObservedRunningTime="2026-07-24 22:28:34.713235534 +0000 UTC m=+4395.906413475"
Jul 24 22:28:35.691064 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:35.691030    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:35.692473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:35.692442    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:28:36.695455 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:36.695413    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:28:41.700003 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:41.699925    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:28:41.700557 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:41.700467    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:28:51.701166 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:28:51.701128    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:29:01.700367 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:01.700327    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:29:11.701378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:11.701340    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:29:21.701190 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:21.701145    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:29:31.701221 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:31.701184    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:29:41.700866 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:41.700827    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:29:51.701375 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:51.701339    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:29:58.573359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:58.573327    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"]
Jul 24 22:29:58.573910 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:58.573614    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" containerID="cri-o://935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4" gracePeriod=30
Jul 24 22:29:58.573910 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:58.573640    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kube-rbac-proxy" containerID="cri-o://8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e" gracePeriod=30
Jul 24 22:29:58.590394 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:58.590372    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:29:58.993032 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:58.993001    2585 generic.go:356] "Generic (PLEG): container finished" podID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerID="8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e" exitCode=2
Jul 24 22:29:58.993192 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:29:58.993075    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerDied","Data":"8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e"}
Jul 24 22:30:01.695859 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:01.695814    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 24 22:30:01.701148 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:01.701123    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 24 22:30:02.917864 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:02.917838    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:30:03.009786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.009716    2585 generic.go:356] "Generic (PLEG): container finished" podID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerID="935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4" exitCode=0
Jul 24 22:30:03.009786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.009770    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"
Jul 24 22:30:03.009929 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.009800    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerDied","Data":"935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4"}
Jul 24 22:30:03.009929 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.009866    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d" event={"ID":"77bc93e9-85c4-4c09-94ba-9444aae93411","Type":"ContainerDied","Data":"07430f10ff6171179285bef19c1e6bfedff3956be12831efd9ac2f45c81ef2d1"}
Jul 24 22:30:03.009929 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.009895    2585 scope.go:117] "RemoveContainer" containerID="8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e"
Jul 24 22:30:03.017741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.017721    2585 scope.go:117] "RemoveContainer" containerID="935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4"
Jul 24 22:30:03.024531 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.024515    2585 scope.go:117] "RemoveContainer" containerID="5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c"
Jul 24 22:30:03.031887 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.031869    2585 scope.go:117] "RemoveContainer" containerID="8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e"
Jul 24 22:30:03.032154 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:30:03.032129    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e\": container with ID starting with 8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e not found: ID does not exist" containerID="8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e"
Jul 24 22:30:03.032231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.032160    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e"} err="failed to get container status \"8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e\": rpc error: code = NotFound desc = could not find container \"8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e\": container with ID starting with 8fc1a05b20ce6a139dd1fb5d6e859ed584759659cd3ae6a10e18a1d69db9fa4e not found: ID does not exist"
Jul 24 22:30:03.032231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.032185    2585 scope.go:117] "RemoveContainer" containerID="935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4"
Jul 24 22:30:03.032439 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:30:03.032421    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4\": container with ID starting with 935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4 not found: ID does not exist" containerID="935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4"
Jul 24 22:30:03.032478 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.032446    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4"} err="failed to get container status \"935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4\": rpc error: code = NotFound desc = could not find container \"935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4\": container with ID starting with 935686c2f9159135262abfaa5df5912bb5f2adf952b9255fedc5a391cddb56c4 not found: ID does not exist"
Jul 24 22:30:03.032478 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.032461    2585 scope.go:117] "RemoveContainer" containerID="5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c"
Jul 24 22:30:03.032704 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:30:03.032686    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c\": container with ID starting with 5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c not found: ID does not exist" containerID="5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c"
Jul 24 22:30:03.032756 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.032709    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c"} err="failed to get container status \"5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c\": rpc error: code = NotFound desc = could not find container \"5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c\": container with ID starting with 5aed17fb80dfac2e9c045fc5ee6ed6cbee70b2f1be853d867fb76beca8cbd38c not found: ID does not exist"
Jul 24 22:30:03.056439 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056422    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls\") pod \"77bc93e9-85c4-4c09-94ba-9444aae93411\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") "
Jul 24 22:30:03.056520 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056447    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-cabundle-cert\") pod \"77bc93e9-85c4-4c09-94ba-9444aae93411\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") "
Jul 24 22:30:03.056520 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056500    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs4q4\" (UniqueName: \"kubernetes.io/projected/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-api-access-gs4q4\") pod \"77bc93e9-85c4-4c09-94ba-9444aae93411\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") "
Jul 24 22:30:03.056644 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056624    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77bc93e9-85c4-4c09-94ba-9444aae93411-kserve-provision-location\") pod \"77bc93e9-85c4-4c09-94ba-9444aae93411\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") "
Jul 24 22:30:03.056718 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056704    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-rbac-proxy-sar-config\") pod \"77bc93e9-85c4-4c09-94ba-9444aae93411\" (UID: \"77bc93e9-85c4-4c09-94ba-9444aae93411\") "
Jul 24 22:30:03.056898 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056877    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77bc93e9-85c4-4c09-94ba-9444aae93411" (UID: "77bc93e9-85c4-4c09-94ba-9444aae93411"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:30:03.056970 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.056924    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bc93e9-85c4-4c09-94ba-9444aae93411-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77bc93e9-85c4-4c09-94ba-9444aae93411" (UID: "77bc93e9-85c4-4c09-94ba-9444aae93411"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:30:03.057045 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.057022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77bc93e9-85c4-4c09-94ba-9444aae93411" (UID: "77bc93e9-85c4-4c09-94ba-9444aae93411"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:30:03.059026 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.059009    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77bc93e9-85c4-4c09-94ba-9444aae93411" (UID: "77bc93e9-85c4-4c09-94ba-9444aae93411"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:30:03.059083 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.059028    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-api-access-gs4q4" (OuterVolumeSpecName: "kube-api-access-gs4q4") pod "77bc93e9-85c4-4c09-94ba-9444aae93411" (UID: "77bc93e9-85c4-4c09-94ba-9444aae93411"). InnerVolumeSpecName "kube-api-access-gs4q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:30:03.157777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.157746    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77bc93e9-85c4-4c09-94ba-9444aae93411-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:30:03.157777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.157776    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:30:03.157777 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.157786    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs4q4\" (UniqueName: \"kubernetes.io/projected/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-api-access-gs4q4\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:30:03.157933 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.157794    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77bc93e9-85c4-4c09-94ba-9444aae93411-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:30:03.157933 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.157803    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77bc93e9-85c4-4c09-94ba-9444aae93411-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:30:03.335627 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.335602    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"]
Jul 24 22:30:03.339490 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.339464    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7ddbdd7546-g7n6d"]
Jul 24 22:30:03.676484 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.676448    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:30:03.857765 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:03.857719    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:30:05.327216 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:05.327182    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" path="/var/lib/kubelet/pods/77bc93e9-85c4-4c09-94ba-9444aae93411/volumes"
Jul 24 22:30:44.881831 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:44.881794    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:30:50.009688 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.009627    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:30:50.050032 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.049992    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"]
Jul 24 22:30:50.050324 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050307    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kube-rbac-proxy"
Jul 24 22:30:50.050412 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050327    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kube-rbac-proxy"
Jul 24 22:30:50.050412 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050364    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="storage-initializer"
Jul 24 22:30:50.050412 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050374    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="storage-initializer"
Jul 24 22:30:50.050412 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050389    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container"
Jul 24 22:30:50.050412 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050398    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container"
Jul 24 22:30:50.050687 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050458    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kube-rbac-proxy"
Jul 24 22:30:50.050687 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.050474    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bc93e9-85c4-4c09-94ba-9444aae93411" containerName="kserve-container"
Jul 24 22:30:50.053573 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.053552    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.056407 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.056340    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:30:50.056407 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.056375    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:30:50.056572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.056423    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:30:50.056572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.056423    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:30:50.056572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.056460    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:30:50.057500 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.057467    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:30:50.057608 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.057495    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 24 22:30:50.063264 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.063239    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"]
Jul 24 22:30:50.095555 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.095526    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.095695 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.095562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2fh\" (UniqueName: \"kubernetes.io/projected/80de0783-7981-4bfd-ad4f-208833566eb5-kube-api-access-cl2fh\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.095695 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.095597    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.095830 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.095703    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80de0783-7981-4bfd-ad4f-208833566eb5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.095830 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.095743    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197051 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197023    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197218 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2fh\" (UniqueName: \"kubernetes.io/projected/80de0783-7981-4bfd-ad4f-208833566eb5-kube-api-access-cl2fh\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197218 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197076    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197218 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:30:50.197177    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 24 22:30:50.197218 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197193    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80de0783-7981-4bfd-ad4f-208833566eb5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197483 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197483 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:30:50.197309    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls podName:80de0783-7981-4bfd-ad4f-208833566eb5 nodeName:}" failed. No retries permitted until 2026-07-24 22:30:50.697287716 +0000 UTC m=+4531.890465631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-7699m" (UID: "80de0783-7981-4bfd-ad4f-208833566eb5") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 24 22:30:50.197600 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80de0783-7981-4bfd-ad4f-208833566eb5-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197852 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197827    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.197956 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.197897    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.210490 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.210473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2fh\" (UniqueName: \"kubernetes.io/projected/80de0783-7981-4bfd-ad4f-208833566eb5-kube-api-access-cl2fh\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.701504 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.701462    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.704078 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.704053    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7699m\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:50.965507 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:50.965425    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:30:51.094305 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:51.094281    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"]
Jul 24 22:30:51.096556 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:30:51.096533    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80de0783_7981_4bfd_ad4f_208833566eb5.slice/crio-b3aeea189bc4129e7df0f05677af5e77d73af05e588730a49848142dd1dcd3a6 WatchSource:0}: Error finding container b3aeea189bc4129e7df0f05677af5e77d73af05e588730a49848142dd1dcd3a6: Status 404 returned error can't find the container with id b3aeea189bc4129e7df0f05677af5e77d73af05e588730a49848142dd1dcd3a6
Jul 24 22:30:51.133681 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:51.133638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:30:51.183159 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:51.183130    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerStarted","Data":"a9be7f4774121f247d8da2832c83d51942d7d9a88c5bfb6fe4162bdb4588de4f"}
Jul 24 22:30:51.183271 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:51.183183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerStarted","Data":"b3aeea189bc4129e7df0f05677af5e77d73af05e588730a49848142dd1dcd3a6"}
Jul 24 22:30:56.206836 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:56.206801    2585 generic.go:356] "Generic (PLEG): container finished" podID="80de0783-7981-4bfd-ad4f-208833566eb5" containerID="a9be7f4774121f247d8da2832c83d51942d7d9a88c5bfb6fe4162bdb4588de4f" exitCode=0
Jul 24 22:30:56.207231 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:56.206876    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerDied","Data":"a9be7f4774121f247d8da2832c83d51942d7d9a88c5bfb6fe4162bdb4588de4f"}
Jul 24 22:30:56.208123 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:30:56.208107    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:31:00.225444 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:00.225407    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerStarted","Data":"18ae64c2ea4ef631f3bd40fe2d4465b1991de5e960115d255fcb149f3d8b410b"}
Jul 24 22:31:00.225919 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:00.225493    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerStarted","Data":"fa68ab41e83b277af1a89814074d3afa19d4c2bccd843b5e98260e5cd5434385"}
Jul 24 22:31:00.225919 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:00.225765    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:31:00.226032 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:00.225920    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:31:00.227224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:00.227198    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 24 22:31:01.229913 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:01.229868    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 24 22:31:06.233967 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:06.233937    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:31:06.234482 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:06.234422    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 24 22:31:06.254399 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:06.254355    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podStartSLOduration=8.14444475 podStartE2EDuration="16.254338142s" totalImagesPullingTime="3.640721525s" totalInitContainerRuntime="4.469171867s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:31:00.244164019 +0000 UTC m=+4541.437341958" watchObservedRunningTime="2026-07-24 22:31:06.254338142 +0000 UTC m=+4547.447516078"
Jul 24 22:31:16.235504 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:16.235471    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:31:30.335526 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:30.335497    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"]
Jul 24 22:31:30.336115 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:30.335786    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container" containerID="cri-o://fa68ab41e83b277af1a89814074d3afa19d4c2bccd843b5e98260e5cd5434385" gracePeriod=30
Jul 24 22:31:30.336115 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:30.335935    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" containerID="cri-o://18ae64c2ea4ef631f3bd40fe2d4465b1991de5e960115d255fcb149f3d8b410b" gracePeriod=30
Jul 24 22:31:30.360591 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:30.360556    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:31:31.230878 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:31.230838    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 24 22:31:31.343211 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:31.343180    2585 generic.go:356] "Generic (PLEG): container finished" podID="80de0783-7981-4bfd-ad4f-208833566eb5" containerID="18ae64c2ea4ef631f3bd40fe2d4465b1991de5e960115d255fcb149f3d8b410b" exitCode=2
Jul 24 22:31:31.343577 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:31.343230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerDied","Data":"18ae64c2ea4ef631f3bd40fe2d4465b1991de5e960115d255fcb149f3d8b410b"}
Jul 24 22:31:35.497002 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:35.496952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:31:35.662383 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:35.662347    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:31:36.230254 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:36.230212    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 24 22:31:41.230359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:41.230272    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 24 22:31:41.230833 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:41.230387    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:31:46.230765 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:46.230722    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 24 22:31:51.230982 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:51.230937    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 24 22:31:56.230221 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:31:56.230178    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 24 22:32:00.441719 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:00.441683    2585 generic.go:356] "Generic (PLEG): container finished" podID="80de0783-7981-4bfd-ad4f-208833566eb5" containerID="fa68ab41e83b277af1a89814074d3afa19d4c2bccd843b5e98260e5cd5434385" exitCode=137
Jul 24 22:32:00.442070 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:00.441731    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerDied","Data":"fa68ab41e83b277af1a89814074d3afa19d4c2bccd843b5e98260e5cd5434385"}
Jul 24 22:32:00.972075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:00.972049    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:32:01.113052 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.112970    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-kube-rbac-proxy-sar-config\") pod \"80de0783-7981-4bfd-ad4f-208833566eb5\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") "
Jul 24 22:32:01.113052 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.113017    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80de0783-7981-4bfd-ad4f-208833566eb5-kserve-provision-location\") pod \"80de0783-7981-4bfd-ad4f-208833566eb5\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") "
Jul 24 22:32:01.113283 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.113070    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-cabundle-cert\") pod \"80de0783-7981-4bfd-ad4f-208833566eb5\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") "
Jul 24 22:32:01.113283 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.113124    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls\") pod \"80de0783-7981-4bfd-ad4f-208833566eb5\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") "
Jul 24 22:32:01.113283 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.113156    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl2fh\" (UniqueName: \"kubernetes.io/projected/80de0783-7981-4bfd-ad4f-208833566eb5-kube-api-access-cl2fh\") pod \"80de0783-7981-4bfd-ad4f-208833566eb5\" (UID: \"80de0783-7981-4bfd-ad4f-208833566eb5\") "
Jul 24 22:32:01.113474 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.113446    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80de0783-7981-4bfd-ad4f-208833566eb5" (UID: "80de0783-7981-4bfd-ad4f-208833566eb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:32:01.113563 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.113546    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80de0783-7981-4bfd-ad4f-208833566eb5" (UID: "80de0783-7981-4bfd-ad4f-208833566eb5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:32:01.115459 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.115427    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80de0783-7981-4bfd-ad4f-208833566eb5" (UID: "80de0783-7981-4bfd-ad4f-208833566eb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:32:01.115566 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.115434    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80de0783-7981-4bfd-ad4f-208833566eb5-kube-api-access-cl2fh" (OuterVolumeSpecName: "kube-api-access-cl2fh") pod "80de0783-7981-4bfd-ad4f-208833566eb5" (UID: "80de0783-7981-4bfd-ad4f-208833566eb5"). InnerVolumeSpecName "kube-api-access-cl2fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:32:01.124622 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.124594    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80de0783-7981-4bfd-ad4f-208833566eb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80de0783-7981-4bfd-ad4f-208833566eb5" (UID: "80de0783-7981-4bfd-ad4f-208833566eb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:32:01.213625 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.213590    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:32:01.213625 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.213621    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80de0783-7981-4bfd-ad4f-208833566eb5-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:32:01.213779 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.213633    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80de0783-7981-4bfd-ad4f-208833566eb5-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:32:01.213779 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.213652    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80de0783-7981-4bfd-ad4f-208833566eb5-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:32:01.213779 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.213660    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl2fh\" (UniqueName: \"kubernetes.io/projected/80de0783-7981-4bfd-ad4f-208833566eb5-kube-api-access-cl2fh\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:32:01.446710 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.446686    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"
Jul 24 22:32:01.447126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.446683    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m" event={"ID":"80de0783-7981-4bfd-ad4f-208833566eb5","Type":"ContainerDied","Data":"b3aeea189bc4129e7df0f05677af5e77d73af05e588730a49848142dd1dcd3a6"}
Jul 24 22:32:01.447126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.446822    2585 scope.go:117] "RemoveContainer" containerID="18ae64c2ea4ef631f3bd40fe2d4465b1991de5e960115d255fcb149f3d8b410b"
Jul 24 22:32:01.454880 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.454866    2585 scope.go:117] "RemoveContainer" containerID="fa68ab41e83b277af1a89814074d3afa19d4c2bccd843b5e98260e5cd5434385"
Jul 24 22:32:01.461754 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.461736    2585 scope.go:117] "RemoveContainer" containerID="a9be7f4774121f247d8da2832c83d51942d7d9a88c5bfb6fe4162bdb4588de4f"
Jul 24 22:32:01.465571 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.465551    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"]
Jul 24 22:32:01.469565 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:01.469547    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7699m"]
Jul 24 22:32:03.326503 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:03.326470    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" path="/var/lib/kubelet/pods/80de0783-7981-4bfd-ad4f-208833566eb5/volumes"
Jul 24 22:32:37.243514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:32:37.243478    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:33:42.415264 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:42.415222    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:33:47.917244 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:47.917206    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:33:53.023878 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.023842    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:33:53.086976 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.086943    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"]
Jul 24 22:33:53.087282 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087265    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container"
Jul 24 22:33:53.087364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087285    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container"
Jul 24 22:33:53.087364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087299    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="storage-initializer"
Jul 24 22:33:53.087364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087307    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="storage-initializer"
Jul 24 22:33:53.087364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087330    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy"
Jul 24 22:33:53.087364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087338    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy"
Jul 24 22:33:53.087626 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087410    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kserve-container"
Jul 24 22:33:53.087626 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.087423    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80de0783-7981-4bfd-ad4f-208833566eb5" containerName="kube-rbac-proxy"
Jul 24 22:33:53.090530 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.090511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.096977 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.096950    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 24 22:33:53.097110 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.097010    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:33:53.097179 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.097105    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 24 22:33:53.097179 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.097119    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:33:53.097295 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.097197    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:33:53.098028 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.097993    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:33:53.098136 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.098116    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:33:53.100875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.100854    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"]
Jul 24 22:33:53.117321 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.117294    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tk5\" (UniqueName: \"kubernetes.io/projected/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-api-access-w6tk5\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.117425 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.117326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e102ed0f-0f26-4d86-a63e-979993d6acb2-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.117425 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.117344    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.117425 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.117367    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.117559 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.117425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218046 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218190 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218078    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tk5\" (UniqueName: \"kubernetes.io/projected/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-api-access-w6tk5\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218190 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218110    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e102ed0f-0f26-4d86-a63e-979993d6acb2-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218190 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218190 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218403 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:33:53.218288    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Jul 24 22:33:53.218403 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:33:53.218352    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls podName:e102ed0f-0f26-4d86-a63e-979993d6acb2 nodeName:}" failed. No retries permitted until 2026-07-24 22:33:53.718330933 +0000 UTC m=+4714.911508865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-tc894" (UID: "e102ed0f-0f26-4d86-a63e-979993d6acb2") : secret "isvc-xgboost-predictor-serving-cert" not found
Jul 24 22:33:53.218813 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218782    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e102ed0f-0f26-4d86-a63e-979993d6acb2-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218813 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218784    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.218957 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.218838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.229023 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.228997    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tk5\" (UniqueName: \"kubernetes.io/projected/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-api-access-w6tk5\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.374928 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.374897    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:33:53.721118 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.721041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:53.723562 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:53.723537    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-tc894\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:54.007649 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:54.007568    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:33:54.134688 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:54.134645    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"]
Jul 24 22:33:54.136837 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:33:54.136812    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode102ed0f_0f26_4d86_a63e_979993d6acb2.slice/crio-6622eb15e559b7f555c40b7bad818eeb384c27f41bb098cd5002e7d77e9deeb1 WatchSource:0}: Error finding container 6622eb15e559b7f555c40b7bad818eeb384c27f41bb098cd5002e7d77e9deeb1: Status 404 returned error can't find the container with id 6622eb15e559b7f555c40b7bad818eeb384c27f41bb098cd5002e7d77e9deeb1
Jul 24 22:33:54.857119 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:54.857086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerStarted","Data":"c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6"}
Jul 24 22:33:54.857304 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:54.857141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerStarted","Data":"6622eb15e559b7f555c40b7bad818eeb384c27f41bb098cd5002e7d77e9deeb1"}
Jul 24 22:33:58.875825 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:58.875786    2585 generic.go:356] "Generic (PLEG): container finished" podID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerID="c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6" exitCode=0
Jul 24 22:33:58.876424 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:33:58.876393    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerDied","Data":"c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6"}
Jul 24 22:34:16.954713 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:16.954657    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerStarted","Data":"226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a"}
Jul 24 22:34:16.955221 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:16.954753    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerStarted","Data":"2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9"}
Jul 24 22:34:16.955221 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:16.955075    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:34:16.982000 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:16.981943    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podStartSLOduration=2.78686918 podStartE2EDuration="23.981927646s" totalImagesPullingTime="17.279709063s" totalInitContainerRuntime="3.915349403s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:34:16.980461575 +0000 UTC m=+4738.173639513" watchObservedRunningTime="2026-07-24 22:34:16.981927646 +0000 UTC m=+4738.175105582"
Jul 24 22:34:17.959594 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:17.959553    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:34:17.961073 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:17.961046    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:34:18.963654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:18.963617    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:34:23.970178 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:23.970140    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:34:23.970781 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:23.970749    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:34:33.971290 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:33.971250    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:34:43.970866 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:43.970818    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:34:53.970849 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:34:53.970806    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:35:03.971228 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:03.971188    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:35:13.972541 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:13.972508    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:35:23.203438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:23.203399    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"]
Jul 24 22:35:23.203980 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:23.203676    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" containerID="cri-o://2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9" gracePeriod=30
Jul 24 22:35:23.203980 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:23.203688    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kube-rbac-proxy" containerID="cri-o://226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a" gracePeriod=30
Jul 24 22:35:23.218475 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:23.218450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:35:23.964617 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:23.964570    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 24 22:35:23.970952 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:23.970929    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 24 22:35:24.194723 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:24.194682    2585 generic.go:356] "Generic (PLEG): container finished" podID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerID="226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a" exitCode=2
Jul 24 22:35:24.194723 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:24.194700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerDied","Data":"226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a"}
Jul 24 22:35:26.446682 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.446648    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:35:26.507876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.507839    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e102ed0f-0f26-4d86-a63e-979993d6acb2-kserve-provision-location\") pod \"e102ed0f-0f26-4d86-a63e-979993d6acb2\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") "
Jul 24 22:35:26.507876 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.507884    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6tk5\" (UniqueName: \"kubernetes.io/projected/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-api-access-w6tk5\") pod \"e102ed0f-0f26-4d86-a63e-979993d6acb2\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") "
Jul 24 22:35:26.508114 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.507908    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-cabundle-cert\") pod \"e102ed0f-0f26-4d86-a63e-979993d6acb2\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") "
Jul 24 22:35:26.508114 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.507955    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-rbac-proxy-sar-config\") pod \"e102ed0f-0f26-4d86-a63e-979993d6acb2\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") "
Jul 24 22:35:26.508114 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.508036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls\") pod \"e102ed0f-0f26-4d86-a63e-979993d6acb2\" (UID: \"e102ed0f-0f26-4d86-a63e-979993d6acb2\") "
Jul 24 22:35:26.508276 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.508227    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e102ed0f-0f26-4d86-a63e-979993d6acb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e102ed0f-0f26-4d86-a63e-979993d6acb2" (UID: "e102ed0f-0f26-4d86-a63e-979993d6acb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:35:26.508353 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.508329    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e102ed0f-0f26-4d86-a63e-979993d6acb2" (UID: "e102ed0f-0f26-4d86-a63e-979993d6acb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:35:26.508390 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.508335    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e102ed0f-0f26-4d86-a63e-979993d6acb2" (UID: "e102ed0f-0f26-4d86-a63e-979993d6acb2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:35:26.510121 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.510101    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-api-access-w6tk5" (OuterVolumeSpecName: "kube-api-access-w6tk5") pod "e102ed0f-0f26-4d86-a63e-979993d6acb2" (UID: "e102ed0f-0f26-4d86-a63e-979993d6acb2"). InnerVolumeSpecName "kube-api-access-w6tk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:35:26.510193 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.510154    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e102ed0f-0f26-4d86-a63e-979993d6acb2" (UID: "e102ed0f-0f26-4d86-a63e-979993d6acb2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:35:26.608827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.608763    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:35:26.608827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.608793    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e102ed0f-0f26-4d86-a63e-979993d6acb2-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:35:26.608827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.608807    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e102ed0f-0f26-4d86-a63e-979993d6acb2-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:35:26.608827 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.608821    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tk5\" (UniqueName: \"kubernetes.io/projected/e102ed0f-0f26-4d86-a63e-979993d6acb2-kube-api-access-w6tk5\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:35:26.609038 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:26.608836    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e102ed0f-0f26-4d86-a63e-979993d6acb2-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:35:27.207218 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.207186    2585 generic.go:356] "Generic (PLEG): container finished" podID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerID="2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9" exitCode=0
Jul 24 22:35:27.207398 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.207246    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"
Jul 24 22:35:27.207398 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.207249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerDied","Data":"2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9"}
Jul 24 22:35:27.207398 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.207379    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894" event={"ID":"e102ed0f-0f26-4d86-a63e-979993d6acb2","Type":"ContainerDied","Data":"6622eb15e559b7f555c40b7bad818eeb384c27f41bb098cd5002e7d77e9deeb1"}
Jul 24 22:35:27.207535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.207407    2585 scope.go:117] "RemoveContainer" containerID="226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a"
Jul 24 22:35:27.216286 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.216266    2585 scope.go:117] "RemoveContainer" containerID="2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9"
Jul 24 22:35:27.223564 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.223548    2585 scope.go:117] "RemoveContainer" containerID="c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6"
Jul 24 22:35:27.230626 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.230606    2585 scope.go:117] "RemoveContainer" containerID="226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a"
Jul 24 22:35:27.230895 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:35:27.230875    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a\": container with ID starting with 226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a not found: ID does not exist" containerID="226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a"
Jul 24 22:35:27.230965 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.230902    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a"} err="failed to get container status \"226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a\": rpc error: code = NotFound desc = could not find container \"226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a\": container with ID starting with 226fd0cc56b3958a102edd7253ed35f57af26525b186dd29d12e5a18180cc54a not found: ID does not exist"
Jul 24 22:35:27.230965 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.230918    2585 scope.go:117] "RemoveContainer" containerID="2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9"
Jul 24 22:35:27.231127 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:35:27.231112    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9\": container with ID starting with 2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9 not found: ID does not exist" containerID="2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9"
Jul 24 22:35:27.231175 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.231130    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9"} err="failed to get container status \"2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9\": rpc error: code = NotFound desc = could not find container \"2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9\": container with ID starting with 2d4719441be4e9b696312b8400cce2107c03602c0f318a9980cdc7f940b353d9 not found: ID does not exist"
Jul 24 22:35:27.231175 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.231142    2585 scope.go:117] "RemoveContainer" containerID="c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6"
Jul 24 22:35:27.231323 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:35:27.231307    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6\": container with ID starting with c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6 not found: ID does not exist" containerID="c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6"
Jul 24 22:35:27.231366 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.231325    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6"} err="failed to get container status \"c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6\": rpc error: code = NotFound desc = could not find container \"c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6\": container with ID starting with c3c4f4f5b4ed2722f9c04456868939b6f650495a8c331f90d5ea7b3b409a4ba6 not found: ID does not exist"
Jul 24 22:35:27.248469 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.248446    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"]
Jul 24 22:35:27.255323 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.255298    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-tc894"]
Jul 24 22:35:27.326836 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:27.326809    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" path="/var/lib/kubelet/pods/e102ed0f-0f26-4d86-a63e-979993d6acb2/volumes"
Jul 24 22:35:28.314081 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.314046    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:35:28.362538 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.362504    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"]
Jul 24 22:35:28.362940 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.362919    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container"
Jul 24 22:35:28.362940 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.362939    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container"
Jul 24 22:35:28.363116 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.362958    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="storage-initializer"
Jul 24 22:35:28.363116 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.362967    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="storage-initializer"
Jul 24 22:35:28.363116 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.362993    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kube-rbac-proxy"
Jul 24 22:35:28.363116 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.363001    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kube-rbac-proxy"
Jul 24 22:35:28.363116 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.363097    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kserve-container"
Jul 24 22:35:28.363116 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.363108    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e102ed0f-0f26-4d86-a63e-979993d6acb2" containerName="kube-rbac-proxy"
Jul 24 22:35:28.365901 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.365883    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.368392 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.368365    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 24 22:35:28.368524 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.368439    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 24 22:35:28.368524 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.368459    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:35:28.368760 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.368744    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:35:28.368875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.368826    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:35:28.368930 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.368874    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:35:28.369685 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.369650    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:35:28.375447 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.375424    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"]
Jul 24 22:35:28.419857 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.419828    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ced53a-f351-4fc8-ac46-ac0b12b34824-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.419978 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.419859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmn5\" (UniqueName: \"kubernetes.io/projected/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-api-access-tvmn5\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.419978 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.419884    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.419978 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.419914    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.420140 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.419974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.520531 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.520499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ced53a-f351-4fc8-ac46-ac0b12b34824-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.520531 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.520534    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmn5\" (UniqueName: \"kubernetes.io/projected/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-api-access-tvmn5\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.520791 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.520561    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.520791 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.520711    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.520791 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.520749    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.520963 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:35:28.520924    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 24 22:35:28.521018 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.520962    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ced53a-f351-4fc8-ac46-ac0b12b34824-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.521018 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:35:28.521003    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls podName:80ced53a-f351-4fc8-ac46-ac0b12b34824 nodeName:}" failed. No retries permitted until 2026-07-24 22:35:29.02098541 +0000 UTC m=+4810.214163339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" (UID: "80ced53a-f351-4fc8-ac46-ac0b12b34824") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 24 22:35:28.521250 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.521230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.521374 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.521326    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:28.530095 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:28.530073    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmn5\" (UniqueName: \"kubernetes.io/projected/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-api-access-tvmn5\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:29.001354 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:29.001317    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:35:29.024791 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:29.024767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:29.027311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:29.027294    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:29.277893 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:29.277797    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:29.409174 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:29.409137    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"]
Jul 24 22:35:29.414751 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:35:29.414726    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ced53a_f351_4fc8_ac46_ac0b12b34824.slice/crio-8a33566ed62f47c5884e9d420bfcf49860d1adc6638ba4c42b5bd1fea95ad6e4 WatchSource:0}: Error finding container 8a33566ed62f47c5884e9d420bfcf49860d1adc6638ba4c42b5bd1fea95ad6e4: Status 404 returned error can't find the container with id 8a33566ed62f47c5884e9d420bfcf49860d1adc6638ba4c42b5bd1fea95ad6e4
Jul 24 22:35:30.221081 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:30.221043    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerStarted","Data":"8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21"}
Jul 24 22:35:30.221303 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:30.221099    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerStarted","Data":"8a33566ed62f47c5884e9d420bfcf49860d1adc6638ba4c42b5bd1fea95ad6e4"}
Jul 24 22:35:34.237183 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:34.237149    2585 generic.go:356] "Generic (PLEG): container finished" podID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerID="8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21" exitCode=0
Jul 24 22:35:34.237578 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:34.237220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerDied","Data":"8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21"}
Jul 24 22:35:35.243432 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:35.243397    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerStarted","Data":"757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb"}
Jul 24 22:35:35.243849 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:35.243465    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerStarted","Data":"5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096"}
Jul 24 22:35:35.243849 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:35.243643    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:35.265364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:35.265319    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" podStartSLOduration=3.362427284 podStartE2EDuration="7.265305163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.902877879s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:35:35.263370705 +0000 UTC m=+4816.456548663" watchObservedRunningTime="2026-07-24 22:35:35.265305163 +0000 UTC m=+4816.458483099"
Jul 24 22:35:36.247175 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:36.247131    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:35:42.257307 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:35:42.257278    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:36:12.262090 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:12.262005    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:36:18.443543 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:18.443514    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"]
Jul 24 22:36:18.444011 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:18.443763    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kserve-container" containerID="cri-o://5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096" gracePeriod=30
Jul 24 22:36:18.444011 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:18.443818    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kube-rbac-proxy" containerID="cri-o://757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb" gracePeriod=30
Jul 24 22:36:18.462599 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:18.462538    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:36:19.399997 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:19.399967    2585 generic.go:356] "Generic (PLEG): container finished" podID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerID="757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb" exitCode=2
Jul 24 22:36:19.400174 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:19.400043    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerDied","Data":"757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb"}
Jul 24 22:36:22.252506 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:22.252463    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 24 22:36:22.257763 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:22.257736    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 24 22:36:23.589001 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.588950    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:36:23.614550 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.614519    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"]
Jul 24 22:36:23.619141 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.619114    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.621791 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.621773    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 24 22:36:23.621891 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.621804    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 24 22:36:23.628705 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.628680    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"]
Jul 24 22:36:23.693972 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.693940    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:36:23.729751 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.729722    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.729897 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.729758    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfcck\" (UniqueName: \"kubernetes.io/projected/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-api-access-jfcck\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.729897 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.729788    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.729897 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.729865    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.730010 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.729932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831162 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831195    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfcck\" (UniqueName: \"kubernetes.io/projected/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-api-access-jfcck\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831421 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831314    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831421 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831356    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831586 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831566    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.831879 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.831850    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.832043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.832026    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.834010 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.833990    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.839981 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.839927    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfcck\" (UniqueName: \"kubernetes.io/projected/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-api-access-jfcck\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-pztsc\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:23.932452 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:23.932419    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:24.060612 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.060571    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"]
Jul 24 22:36:24.062978 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:36:24.062941    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed6bd9b_16af_4a5b_939b_ee2fa11ce93c.slice/crio-2c4629997ec00231cd2cec9b226919d582ed97866d0f58e3230a277954fdb7e6 WatchSource:0}: Error finding container 2c4629997ec00231cd2cec9b226919d582ed97866d0f58e3230a277954fdb7e6: Status 404 returned error can't find the container with id 2c4629997ec00231cd2cec9b226919d582ed97866d0f58e3230a277954fdb7e6
Jul 24 22:36:24.277640 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.277611    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:36:24.419854 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.419745    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerStarted","Data":"de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6"}
Jul 24 22:36:24.419854 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.419825    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerStarted","Data":"2c4629997ec00231cd2cec9b226919d582ed97866d0f58e3230a277954fdb7e6"}
Jul 24 22:36:24.421719 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.421695    2585 generic.go:356] "Generic (PLEG): container finished" podID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerID="5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096" exitCode=0
Jul 24 22:36:24.421861 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.421734    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerDied","Data":"5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096"}
Jul 24 22:36:24.421861 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.421774    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh" event={"ID":"80ced53a-f351-4fc8-ac46-ac0b12b34824","Type":"ContainerDied","Data":"8a33566ed62f47c5884e9d420bfcf49860d1adc6638ba4c42b5bd1fea95ad6e4"}
Jul 24 22:36:24.421861 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.421800    2585 scope.go:117] "RemoveContainer" containerID="757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb"
Jul 24 22:36:24.421861 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.421817    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"
Jul 24 22:36:24.429593 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.429578    2585 scope.go:117] "RemoveContainer" containerID="5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096"
Jul 24 22:36:24.435682 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.435645    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls\") pod \"80ced53a-f351-4fc8-ac46-ac0b12b34824\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") "
Jul 24 22:36:24.435782 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.435746    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvmn5\" (UniqueName: \"kubernetes.io/projected/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-api-access-tvmn5\") pod \"80ced53a-f351-4fc8-ac46-ac0b12b34824\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") "
Jul 24 22:36:24.435848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.435784    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-cabundle-cert\") pod \"80ced53a-f351-4fc8-ac46-ac0b12b34824\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") "
Jul 24 22:36:24.435848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.435816    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ced53a-f351-4fc8-ac46-ac0b12b34824-kserve-provision-location\") pod \"80ced53a-f351-4fc8-ac46-ac0b12b34824\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") "
Jul 24 22:36:24.435848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.435841    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-rbac-proxy-sar-config\") pod \"80ced53a-f351-4fc8-ac46-ac0b12b34824\" (UID: \"80ced53a-f351-4fc8-ac46-ac0b12b34824\") "
Jul 24 22:36:24.436229 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.436202    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80ced53a-f351-4fc8-ac46-ac0b12b34824" (UID: "80ced53a-f351-4fc8-ac46-ac0b12b34824"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:36:24.436480 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.436224    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ced53a-f351-4fc8-ac46-ac0b12b34824-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80ced53a-f351-4fc8-ac46-ac0b12b34824" (UID: "80ced53a-f351-4fc8-ac46-ac0b12b34824"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:36:24.436480 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.436289    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80ced53a-f351-4fc8-ac46-ac0b12b34824" (UID: "80ced53a-f351-4fc8-ac46-ac0b12b34824"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:36:24.437514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.437496    2585 scope.go:117] "RemoveContainer" containerID="8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21"
Jul 24 22:36:24.438517 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.438425    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80ced53a-f351-4fc8-ac46-ac0b12b34824" (UID: "80ced53a-f351-4fc8-ac46-ac0b12b34824"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:36:24.438517 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.438453    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-api-access-tvmn5" (OuterVolumeSpecName: "kube-api-access-tvmn5") pod "80ced53a-f351-4fc8-ac46-ac0b12b34824" (UID: "80ced53a-f351-4fc8-ac46-ac0b12b34824"). InnerVolumeSpecName "kube-api-access-tvmn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:36:24.444654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.444636    2585 scope.go:117] "RemoveContainer" containerID="757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb"
Jul 24 22:36:24.444943 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:36:24.444924    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb\": container with ID starting with 757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb not found: ID does not exist" containerID="757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb"
Jul 24 22:36:24.445007 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.444949    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb"} err="failed to get container status \"757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb\": rpc error: code = NotFound desc = could not find container \"757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb\": container with ID starting with 757c510d0f783355a28f20ceb98dc897cf17a1b776125d42829015e44e8c69eb not found: ID does not exist"
Jul 24 22:36:24.445007 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.444967    2585 scope.go:117] "RemoveContainer" containerID="5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096"
Jul 24 22:36:24.445175 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:36:24.445152    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096\": container with ID starting with 5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096 not found: ID does not exist" containerID="5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096"
Jul 24 22:36:24.445246 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.445186    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096"} err="failed to get container status \"5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096\": rpc error: code = NotFound desc = could not find container \"5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096\": container with ID starting with 5366d5a02675d016880acf5e820101e01b1a04434df7ecfc2fcfe63c9fe6d096 not found: ID does not exist"
Jul 24 22:36:24.445246 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.445209    2585 scope.go:117] "RemoveContainer" containerID="8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21"
Jul 24 22:36:24.445482 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:36:24.445465    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21\": container with ID starting with 8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21 not found: ID does not exist" containerID="8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21"
Jul 24 22:36:24.445519 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.445486    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21"} err="failed to get container status \"8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21\": rpc error: code = NotFound desc = could not find container \"8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21\": container with ID starting with 8ba148084e0e07ebed4db25f38c0c54a9c642748ba853e2212b166aca8bbcc21 not found: ID does not exist"
Jul 24 22:36:24.537012 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.536981    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:36:24.537012 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.537012    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ced53a-f351-4fc8-ac46-ac0b12b34824-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:36:24.537226 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.537025    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:36:24.537226 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.537038    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ced53a-f351-4fc8-ac46-ac0b12b34824-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:36:24.537226 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.537051    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmn5\" (UniqueName: \"kubernetes.io/projected/80ced53a-f351-4fc8-ac46-ac0b12b34824-kube-api-access-tvmn5\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:36:24.744987 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.744942    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"]
Jul 24 22:36:24.749749 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:24.749723    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-fm5bh"]
Jul 24 22:36:25.326499 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:25.326457    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" path="/var/lib/kubelet/pods/80ced53a-f351-4fc8-ac46-ac0b12b34824/volumes"
Jul 24 22:36:28.440204 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:28.440121    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerID="de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6" exitCode=0
Jul 24 22:36:28.440576 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:28.440198    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerDied","Data":"de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6"}
Jul 24 22:36:29.445859 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:29.445827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerStarted","Data":"97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c"}
Jul 24 22:36:29.446229 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:29.445886    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerStarted","Data":"1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946"}
Jul 24 22:36:29.446229 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:29.446098    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:29.468495 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:29.468449    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" podStartSLOduration=2.494019152 podStartE2EDuration="6.468432372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.97441322s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:36:29.466854646 +0000 UTC m=+4870.660032575" watchObservedRunningTime="2026-07-24 22:36:29.468432372 +0000 UTC m=+4870.661610308"
Jul 24 22:36:30.449351 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:30.449319    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:36.459114 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:36.459084    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:36:56.184881 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:36:56.184844    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:37:06.463270 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:06.463236    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:37:13.730178 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:13.730144    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"]
Jul 24 22:37:13.730691 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:13.730480    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kserve-container" containerID="cri-o://1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946" gracePeriod=30
Jul 24 22:37:13.730691 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:13.730487    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kube-rbac-proxy" containerID="cri-o://97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c" gracePeriod=30
Jul 24 22:37:13.741157 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:13.741132    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:37:14.616434 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:14.616404    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerID="97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c" exitCode=2
Jul 24 22:37:14.616614 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:14.616455    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerDied","Data":"97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c"}
Jul 24 22:37:16.453710 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:16.453654    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 24 22:37:16.460335 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:16.460309    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 24 22:37:18.854525 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.854493    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:37:18.894971 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.894937    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"]
Jul 24 22:37:18.895291 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895275    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kserve-container"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895293    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kserve-container"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895307    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kube-rbac-proxy"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895313    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kube-rbac-proxy"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895321    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="storage-initializer"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895326    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="storage-initializer"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895370    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kserve-container"
Jul 24 22:37:18.895391 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.895382    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ced53a-f351-4fc8-ac46-ac0b12b34824" containerName="kube-rbac-proxy"
Jul 24 22:37:18.898298 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.898280    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:18.900822 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.900800    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 24 22:37:18.900822 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.900810    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:37:18.908737 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.908716    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"]
Jul 24 22:37:18.938547 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.938525    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:37:18.999539 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.999511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96j2f\" (UniqueName: \"kubernetes.io/projected/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-api-access-96j2f\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:18.999539 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.999543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/684cd1ca-a09a-48bc-a903-0f645a73aa86-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:18.999759 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.999566    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684cd1ca-a09a-48bc-a903-0f645a73aa86-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:18.999759 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.999584    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:18.999759 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:18.999603    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.100502 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.100475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96j2f\" (UniqueName: \"kubernetes.io/projected/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-api-access-96j2f\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.100660 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.100509    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/684cd1ca-a09a-48bc-a903-0f645a73aa86-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.100660 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.100530    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684cd1ca-a09a-48bc-a903-0f645a73aa86-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.100660 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.100547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.100660 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.100564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.101000 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.100980    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684cd1ca-a09a-48bc-a903-0f645a73aa86-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.101238 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.101221    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.101283 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.101266    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.103156 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.103140    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/684cd1ca-a09a-48bc-a903-0f645a73aa86-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.109315 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.109248    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96j2f\" (UniqueName: \"kubernetes.io/projected/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-api-access-96j2f\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-bwgjp\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.208858 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.208831    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:19.337918 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.337760    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"]
Jul 24 22:37:19.339996 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:37:19.339970    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684cd1ca_a09a_48bc_a903_0f645a73aa86.slice/crio-7efc550f4f01dfeb40d48ab583d6f2df8d9c9238dec29e1dd451314501d7da8e WatchSource:0}: Error finding container 7efc550f4f01dfeb40d48ab583d6f2df8d9c9238dec29e1dd451314501d7da8e: Status 404 returned error can't find the container with id 7efc550f4f01dfeb40d48ab583d6f2df8d9c9238dec29e1dd451314501d7da8e
Jul 24 22:37:19.462745 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.462719    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:37:19.503893 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.503856    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kserve-provision-location\") pod \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") "
Jul 24 22:37:19.503893 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.503887    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfcck\" (UniqueName: \"kubernetes.io/projected/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-api-access-jfcck\") pod \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") "
Jul 24 22:37:19.504126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.503907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-proxy-tls\") pod \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") "
Jul 24 22:37:19.504126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.503947    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-rbac-proxy-sar-config\") pod \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") "
Jul 24 22:37:19.504126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.503974    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-cabundle-cert\") pod \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\" (UID: \"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c\") "
Jul 24 22:37:19.504311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.504220    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" (UID: "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:37:19.504311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.504272    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" (UID: "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:37:19.504505 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.504478    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" (UID: "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:37:19.504637 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.504580    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:37:19.504637 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.504599    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:37:19.504637 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.504613    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:37:19.506751 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.506727    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" (UID: "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:37:19.506837 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.506768    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-api-access-jfcck" (OuterVolumeSpecName: "kube-api-access-jfcck") pod "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" (UID: "8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c"). InnerVolumeSpecName "kube-api-access-jfcck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:37:19.605839 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.605812    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfcck\" (UniqueName: \"kubernetes.io/projected/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-kube-api-access-jfcck\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:37:19.605839 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.605834    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:37:19.636127 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.636054    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerID="1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946" exitCode=0
Jul 24 22:37:19.636127 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.636111    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"
Jul 24 22:37:19.636296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.636119    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerDied","Data":"1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946"}
Jul 24 22:37:19.636296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.636230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc" event={"ID":"8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c","Type":"ContainerDied","Data":"2c4629997ec00231cd2cec9b226919d582ed97866d0f58e3230a277954fdb7e6"}
Jul 24 22:37:19.636296 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.636254    2585 scope.go:117] "RemoveContainer" containerID="97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c"
Jul 24 22:37:19.638071 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.638047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerStarted","Data":"bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba"}
Jul 24 22:37:19.638184 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.638131    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerStarted","Data":"7efc550f4f01dfeb40d48ab583d6f2df8d9c9238dec29e1dd451314501d7da8e"}
Jul 24 22:37:19.644505 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.644487    2585 scope.go:117] "RemoveContainer" containerID="1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946"
Jul 24 22:37:19.652359 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.652340    2585 scope.go:117] "RemoveContainer" containerID="de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6"
Jul 24 22:37:19.660395 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.660359    2585 scope.go:117] "RemoveContainer" containerID="97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c"
Jul 24 22:37:19.660656 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:37:19.660635    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c\": container with ID starting with 97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c not found: ID does not exist" containerID="97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c"
Jul 24 22:37:19.660656 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.660685    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c"} err="failed to get container status \"97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c\": rpc error: code = NotFound desc = could not find container \"97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c\": container with ID starting with 97575c02fa466f06ffabfec6231d44fac7b9af4c05ff192b15af3b2825044a5c not found: ID does not exist"
Jul 24 22:37:19.660656 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.660702    2585 scope.go:117] "RemoveContainer" containerID="1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946"
Jul 24 22:37:19.660926 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:37:19.660915    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946\": container with ID starting with 1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946 not found: ID does not exist" containerID="1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946"
Jul 24 22:37:19.660987 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.660935    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946"} err="failed to get container status \"1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946\": rpc error: code = NotFound desc = could not find container \"1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946\": container with ID starting with 1b5fbafd801a0bc08c91ed469475c1c5c7500d04dd4622620ff993f8cbd1a946 not found: ID does not exist"
Jul 24 22:37:19.660987 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.660950    2585 scope.go:117] "RemoveContainer" containerID="de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6"
Jul 24 22:37:19.661190 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:37:19.661170    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6\": container with ID starting with de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6 not found: ID does not exist" containerID="de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6"
Jul 24 22:37:19.661264 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.661192    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6"} err="failed to get container status \"de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6\": rpc error: code = NotFound desc = could not find container \"de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6\": container with ID starting with de899af85fcefba16c1fb2f0e86055dded053f661299637c30df28a5899484b6 not found: ID does not exist"
Jul 24 22:37:19.682728 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.682701    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"]
Jul 24 22:37:19.685743 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:19.685721    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-pztsc"]
Jul 24 22:37:21.325995 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:21.325965    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" path="/var/lib/kubelet/pods/8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c/volumes"
Jul 24 22:37:23.654642 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:23.654609    2585 generic.go:356] "Generic (PLEG): container finished" podID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerID="bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba" exitCode=0
Jul 24 22:37:23.655029 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:23.654644    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerDied","Data":"bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba"}
Jul 24 22:37:24.661229 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:24.661194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerStarted","Data":"a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5"}
Jul 24 22:37:24.661623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:24.661254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerStarted","Data":"f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb"}
Jul 24 22:37:24.661623 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:24.661445    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:24.683247 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:24.683197    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podStartSLOduration=2.9027950540000003 podStartE2EDuration="6.683183328s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780388274s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:37:24.681854572 +0000 UTC m=+4925.875032573" watchObservedRunningTime="2026-07-24 22:37:24.683183328 +0000 UTC m=+4925.876361264"
Jul 24 22:37:25.665265 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:25.665234    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:25.666511 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:25.666478    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:37:26.668746 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:26.668711    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:37:31.673856 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:31.673826    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:37:31.674339 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:31.674310    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:37:41.675272 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:41.675182    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:37:51.674746 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:37:51.674705    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:38:01.674680 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:01.674620    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:38:11.674742 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:11.674697    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:38:21.675619 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:21.675584    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:38:29.051076 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:29.051039    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"]
Jul 24 22:38:29.051565 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:29.051290    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" containerID="cri-o://f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb" gracePeriod=30
Jul 24 22:38:29.051565 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:29.051390    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kube-rbac-proxy" containerID="cri-o://a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5" gracePeriod=30
Jul 24 22:38:29.061741 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:29.061706    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:38:29.910341 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:29.910307    2585 generic.go:356] "Generic (PLEG): container finished" podID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerID="a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5" exitCode=2
Jul 24 22:38:29.910538 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:29.910379    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerDied","Data":"a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5"}
Jul 24 22:38:31.669252 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:31.669209    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 24 22:38:31.674589 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:31.674558    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 24 22:38:32.397848 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.397819    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:38:32.437414 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437342    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/684cd1ca-a09a-48bc-a903-0f645a73aa86-proxy-tls\") pod \"684cd1ca-a09a-48bc-a903-0f645a73aa86\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") "
Jul 24 22:38:32.437548 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437418    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-cabundle-cert\") pod \"684cd1ca-a09a-48bc-a903-0f645a73aa86\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") "
Jul 24 22:38:32.437548 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437516    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684cd1ca-a09a-48bc-a903-0f645a73aa86-kserve-provision-location\") pod \"684cd1ca-a09a-48bc-a903-0f645a73aa86\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") "
Jul 24 22:38:32.437692 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437559    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96j2f\" (UniqueName: \"kubernetes.io/projected/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-api-access-96j2f\") pod \"684cd1ca-a09a-48bc-a903-0f645a73aa86\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") "
Jul 24 22:38:32.437692 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437595    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-rbac-proxy-sar-config\") pod \"684cd1ca-a09a-48bc-a903-0f645a73aa86\" (UID: \"684cd1ca-a09a-48bc-a903-0f645a73aa86\") "
Jul 24 22:38:32.437862 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437841    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "684cd1ca-a09a-48bc-a903-0f645a73aa86" (UID: "684cd1ca-a09a-48bc-a903-0f645a73aa86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:38:32.437925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.437862    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684cd1ca-a09a-48bc-a903-0f645a73aa86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "684cd1ca-a09a-48bc-a903-0f645a73aa86" (UID: "684cd1ca-a09a-48bc-a903-0f645a73aa86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:38:32.438059 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.438039    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "684cd1ca-a09a-48bc-a903-0f645a73aa86" (UID: "684cd1ca-a09a-48bc-a903-0f645a73aa86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:38:32.439562 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.439541    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684cd1ca-a09a-48bc-a903-0f645a73aa86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "684cd1ca-a09a-48bc-a903-0f645a73aa86" (UID: "684cd1ca-a09a-48bc-a903-0f645a73aa86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:38:32.439766 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.439748    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-api-access-96j2f" (OuterVolumeSpecName: "kube-api-access-96j2f") pod "684cd1ca-a09a-48bc-a903-0f645a73aa86" (UID: "684cd1ca-a09a-48bc-a903-0f645a73aa86"). InnerVolumeSpecName "kube-api-access-96j2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:38:32.538715 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.538684    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:38:32.538715 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.538711    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/684cd1ca-a09a-48bc-a903-0f645a73aa86-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:38:32.538894 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.538726    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96j2f\" (UniqueName: \"kubernetes.io/projected/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-api-access-96j2f\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:38:32.538894 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.538739    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/684cd1ca-a09a-48bc-a903-0f645a73aa86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:38:32.538894 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.538750    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/684cd1ca-a09a-48bc-a903-0f645a73aa86-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:38:32.924310 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.924267    2585 generic.go:356] "Generic (PLEG): container finished" podID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerID="f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb" exitCode=0
Jul 24 22:38:32.924733 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.924326    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"
Jul 24 22:38:32.924733 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.924342    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerDied","Data":"f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb"}
Jul 24 22:38:32.924733 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.924415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp" event={"ID":"684cd1ca-a09a-48bc-a903-0f645a73aa86","Type":"ContainerDied","Data":"7efc550f4f01dfeb40d48ab583d6f2df8d9c9238dec29e1dd451314501d7da8e"}
Jul 24 22:38:32.924733 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.924443    2585 scope.go:117] "RemoveContainer" containerID="a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5"
Jul 24 22:38:32.932729 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.932713    2585 scope.go:117] "RemoveContainer" containerID="f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb"
Jul 24 22:38:32.939795 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.939775    2585 scope.go:117] "RemoveContainer" containerID="bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba"
Jul 24 22:38:32.945491 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.945470    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"]
Jul 24 22:38:32.946979 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.946963    2585 scope.go:117] "RemoveContainer" containerID="a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5"
Jul 24 22:38:32.947216 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:38:32.947194    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5\": container with ID starting with a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5 not found: ID does not exist" containerID="a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5"
Jul 24 22:38:32.947256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.947223    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5"} err="failed to get container status \"a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5\": rpc error: code = NotFound desc = could not find container \"a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5\": container with ID starting with a22f50b85d6a5d259a5301c74f0303cdd475924854836908b9ba8e35e85d3aa5 not found: ID does not exist"
Jul 24 22:38:32.947256 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.947242    2585 scope.go:117] "RemoveContainer" containerID="f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb"
Jul 24 22:38:32.947558 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:38:32.947533    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb\": container with ID starting with f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb not found: ID does not exist" containerID="f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb"
Jul 24 22:38:32.947659 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.947567    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb"} err="failed to get container status \"f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb\": rpc error: code = NotFound desc = could not find container \"f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb\": container with ID starting with f218629ceffc259c096f8e0a4489bdc19185dc2ae85c6a6d150b3112e9c1b5bb not found: ID does not exist"
Jul 24 22:38:32.947659 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.947602    2585 scope.go:117] "RemoveContainer" containerID="bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba"
Jul 24 22:38:32.948100 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:38:32.948021    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba\": container with ID starting with bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba not found: ID does not exist" containerID="bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba"
Jul 24 22:38:32.948100 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.948067    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba"} err="failed to get container status \"bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba\": rpc error: code = NotFound desc = could not find container \"bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba\": container with ID starting with bb3e047335ea244b60f18be5e51c2cc1abfd502f24d5cf3d5e769a27075484ba not found: ID does not exist"
Jul 24 22:38:32.949490 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:32.949468    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-bwgjp"]
Jul 24 22:38:33.326446 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:33.326368    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" path="/var/lib/kubelet/pods/684cd1ca-a09a-48bc-a903-0f645a73aa86/volumes"
Jul 24 22:38:34.132460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.132409    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:38:34.176324 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176287    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"]
Jul 24 22:38:34.176624 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176603    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kserve-container"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176627    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kserve-container"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176644    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kube-rbac-proxy"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176653    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kube-rbac-proxy"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176700    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176709    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176722    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="storage-initializer"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176727    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="storage-initializer"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176734    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kube-rbac-proxy"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176739    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kube-rbac-proxy"
Jul 24 22:38:34.176739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176745    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="storage-initializer"
Jul 24 22:38:34.177233 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176752    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="storage-initializer"
Jul 24 22:38:34.177233 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176812    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kube-rbac-proxy"
Jul 24 22:38:34.177233 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176822    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kube-rbac-proxy"
Jul 24 22:38:34.177233 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176831    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="684cd1ca-a09a-48bc-a903-0f645a73aa86" containerName="kserve-container"
Jul 24 22:38:34.177233 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.176838    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed6bd9b-16af-4a5b-939b-ee2fa11ce93c" containerName="kserve-container"
Jul 24 22:38:34.181102 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.181083    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.184454 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.184425    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 24 22:38:34.184560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.184431    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:38:34.184560 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.184520    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 24 22:38:34.185236 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.185219    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:38:34.185236 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.185225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 22:38:34.185381 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.185258    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:38:34.185381 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.185225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:38:34.191206 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.191181    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"]
Jul 24 22:38:34.243238 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.243209    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:38:34.251371 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.251347    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmq5w\" (UniqueName: \"kubernetes.io/projected/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-api-access-lmq5w\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.251476 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.251386    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.251476 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.251408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366c9d07-29fd-4d3d-a457-d6061740fc60-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.251557 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.251493    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/366c9d07-29fd-4d3d-a457-d6061740fc60-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.251557 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.251526    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.352311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.352286    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.352470 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.352342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmq5w\" (UniqueName: \"kubernetes.io/projected/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-api-access-lmq5w\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.352842 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.352410    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.352996 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.352879    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366c9d07-29fd-4d3d-a457-d6061740fc60-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.353100 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.353053    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/366c9d07-29fd-4d3d-a457-d6061740fc60-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.354008 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.353503    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/366c9d07-29fd-4d3d-a457-d6061740fc60-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.354008 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.353612    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.354008 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.353961    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.357742 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.356809    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366c9d07-29fd-4d3d-a457-d6061740fc60-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.361955 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.361932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmq5w\" (UniqueName: \"kubernetes.io/projected/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-api-access-lmq5w\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.492473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.492410    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:34.619468 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.619443    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"]
Jul 24 22:38:34.621872 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:38:34.621832    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366c9d07_29fd_4d3d_a457_d6061740fc60.slice/crio-1ee20eba95954ef71d501f3e2159e8cfcec57c42f9f01f3a066abb22435703c8 WatchSource:0}: Error finding container 1ee20eba95954ef71d501f3e2159e8cfcec57c42f9f01f3a066abb22435703c8: Status 404 returned error can't find the container with id 1ee20eba95954ef71d501f3e2159e8cfcec57c42f9f01f3a066abb22435703c8
Jul 24 22:38:34.934205 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.934169    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerStarted","Data":"00fa7026acfde76514c083f60d8fbb7f44cd720e872c7011c67d86c9e664761c"}
Jul 24 22:38:34.934398 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:34.934229    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerStarted","Data":"1ee20eba95954ef71d501f3e2159e8cfcec57c42f9f01f3a066abb22435703c8"}
Jul 24 22:38:38.950919 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:38.950882    2585 generic.go:356] "Generic (PLEG): container finished" podID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerID="00fa7026acfde76514c083f60d8fbb7f44cd720e872c7011c67d86c9e664761c" exitCode=0
Jul 24 22:38:38.951311 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:38.950953    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerDied","Data":"00fa7026acfde76514c083f60d8fbb7f44cd720e872c7011c67d86c9e664761c"}
Jul 24 22:38:39.956759 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:39.956725    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerStarted","Data":"73e70f3cd6bed5ace966a3a9b2f062eca7a8e462e3f4e59bea2537cb34f42a3e"}
Jul 24 22:38:39.957196 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:39.956782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerStarted","Data":"118aa509a1b816441e7e14a232fc3d69470374c59d02fb112dbf9d78d6ede07f"}
Jul 24 22:38:39.957196 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:39.956968    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:39.979604 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:39.979559    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" podStartSLOduration=2.468033335 podStartE2EDuration="5.979545308s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.511511973s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:38:39.979138006 +0000 UTC m=+5001.172315955" watchObservedRunningTime="2026-07-24 22:38:39.979545308 +0000 UTC m=+5001.172723273"
Jul 24 22:38:40.960151 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:40.960118    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:46.969110 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:46.969079    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:38:52.510418 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:38:52.510353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:39:16.980957 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:16.980857    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 24 22:39:26.972343 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:26.972309    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:39:34.278630 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:34.278596    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"]
Jul 24 22:39:34.279169 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:34.278857    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kserve-container" containerID="cri-o://118aa509a1b816441e7e14a232fc3d69470374c59d02fb112dbf9d78d6ede07f" gracePeriod=30
Jul 24 22:39:34.279169 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:34.278890    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kube-rbac-proxy" containerID="cri-o://73e70f3cd6bed5ace966a3a9b2f062eca7a8e462e3f4e59bea2537cb34f42a3e" gracePeriod=30
Jul 24 22:39:34.296097 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:34.296075    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:39:35.159083 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:35.159045    2585 generic.go:356] "Generic (PLEG): container finished" podID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerID="73e70f3cd6bed5ace966a3a9b2f062eca7a8e462e3f4e59bea2537cb34f42a3e" exitCode=2
Jul 24 22:39:35.159264 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:35.159121    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerDied","Data":"73e70f3cd6bed5ace966a3a9b2f062eca7a8e462e3f4e59bea2537cb34f42a3e"}
Jul 24 22:39:36.964448 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:36.964403    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 24 22:39:39.387111 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.387058    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:39:39.431005 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.430974    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"]
Jul 24 22:39:39.434776 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.434756    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.437711 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.437683    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 24 22:39:39.437830 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.437815    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 24 22:39:39.446892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.446868    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"]
Jul 24 22:39:39.523264 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.523231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.523430 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.523290    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hl9\" (UniqueName: \"kubernetes.io/projected/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-api-access-m6hl9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.523430 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.523323    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.523430 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.523361    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8fec8e9-a1bc-4080-80ae-26d12497faff-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.523600 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.523440    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8fec8e9-a1bc-4080-80ae-26d12497faff-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.623756 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.623719    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8fec8e9-a1bc-4080-80ae-26d12497faff-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.623913 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.623769    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8fec8e9-a1bc-4080-80ae-26d12497faff-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.623913 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.623807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.623913 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.623852    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hl9\" (UniqueName: \"kubernetes.io/projected/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-api-access-m6hl9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.624075 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.623980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.624190 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.624171    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8fec8e9-a1bc-4080-80ae-26d12497faff-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.624555 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.624525    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.624647 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.624567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.626333 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.626316    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8fec8e9-a1bc-4080-80ae-26d12497faff-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.632595 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.632571    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hl9\" (UniqueName: \"kubernetes.io/projected/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-api-access-m6hl9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-m76lz\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.639031 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.638977    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:39:39.746028 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.745997    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:39.874656 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:39.874634    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"]
Jul 24 22:39:39.876640 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:39:39.876609    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fec8e9_a1bc_4080_80ae_26d12497faff.slice/crio-24ae1487190dd29c87014872ef82d8b4badc64ee386050beeecd3a4d0e5f8fb5 WatchSource:0}: Error finding container 24ae1487190dd29c87014872ef82d8b4badc64ee386050beeecd3a4d0e5f8fb5: Status 404 returned error can't find the container with id 24ae1487190dd29c87014872ef82d8b4badc64ee386050beeecd3a4d0e5f8fb5
Jul 24 22:39:40.182109 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:40.182020    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerStarted","Data":"f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669"}
Jul 24 22:39:40.182109 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:40.182074    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerStarted","Data":"24ae1487190dd29c87014872ef82d8b4badc64ee386050beeecd3a4d0e5f8fb5"}
Jul 24 22:39:41.189063 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.189026    2585 generic.go:356] "Generic (PLEG): container finished" podID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerID="118aa509a1b816441e7e14a232fc3d69470374c59d02fb112dbf9d78d6ede07f" exitCode=0
Jul 24 22:39:41.189448 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.189090    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerDied","Data":"118aa509a1b816441e7e14a232fc3d69470374c59d02fb112dbf9d78d6ede07f"}
Jul 24 22:39:41.226479 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.226452    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:39:41.235284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235236    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmq5w\" (UniqueName: \"kubernetes.io/projected/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-api-access-lmq5w\") pod \"366c9d07-29fd-4d3d-a457-d6061740fc60\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") "
Jul 24 22:39:41.235284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235271    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-rbac-proxy-sar-config\") pod \"366c9d07-29fd-4d3d-a457-d6061740fc60\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") "
Jul 24 22:39:41.235460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235294    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/366c9d07-29fd-4d3d-a457-d6061740fc60-kserve-provision-location\") pod \"366c9d07-29fd-4d3d-a457-d6061740fc60\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") "
Jul 24 22:39:41.235460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235329    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366c9d07-29fd-4d3d-a457-d6061740fc60-proxy-tls\") pod \"366c9d07-29fd-4d3d-a457-d6061740fc60\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") "
Jul 24 22:39:41.235460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235355    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-cabundle-cert\") pod \"366c9d07-29fd-4d3d-a457-d6061740fc60\" (UID: \"366c9d07-29fd-4d3d-a457-d6061740fc60\") "
Jul 24 22:39:41.235723 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235621    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366c9d07-29fd-4d3d-a457-d6061740fc60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "366c9d07-29fd-4d3d-a457-d6061740fc60" (UID: "366c9d07-29fd-4d3d-a457-d6061740fc60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:39:41.235808 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235621    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "366c9d07-29fd-4d3d-a457-d6061740fc60" (UID: "366c9d07-29fd-4d3d-a457-d6061740fc60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:39:41.235872 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.235808    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "366c9d07-29fd-4d3d-a457-d6061740fc60" (UID: "366c9d07-29fd-4d3d-a457-d6061740fc60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:39:41.237458 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.237433    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-api-access-lmq5w" (OuterVolumeSpecName: "kube-api-access-lmq5w") pod "366c9d07-29fd-4d3d-a457-d6061740fc60" (UID: "366c9d07-29fd-4d3d-a457-d6061740fc60"). InnerVolumeSpecName "kube-api-access-lmq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:39:41.237558 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.237494    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366c9d07-29fd-4d3d-a457-d6061740fc60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "366c9d07-29fd-4d3d-a457-d6061740fc60" (UID: "366c9d07-29fd-4d3d-a457-d6061740fc60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:39:41.336764 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.336743    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/366c9d07-29fd-4d3d-a457-d6061740fc60-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:39:41.336764 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.336764    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:39:41.336903 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.336773    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmq5w\" (UniqueName: \"kubernetes.io/projected/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-api-access-lmq5w\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:39:41.336903 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.336783    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/366c9d07-29fd-4d3d-a457-d6061740fc60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:39:41.336903 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:41.336792    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/366c9d07-29fd-4d3d-a457-d6061740fc60-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:39:42.194241 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.194213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp" event={"ID":"366c9d07-29fd-4d3d-a457-d6061740fc60","Type":"ContainerDied","Data":"1ee20eba95954ef71d501f3e2159e8cfcec57c42f9f01f3a066abb22435703c8"}
Jul 24 22:39:42.194604 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.194280    2585 scope.go:117] "RemoveContainer" containerID="73e70f3cd6bed5ace966a3a9b2f062eca7a8e462e3f4e59bea2537cb34f42a3e"
Jul 24 22:39:42.194604 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.194221    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"
Jul 24 22:39:42.202290 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.202274    2585 scope.go:117] "RemoveContainer" containerID="118aa509a1b816441e7e14a232fc3d69470374c59d02fb112dbf9d78d6ede07f"
Jul 24 22:39:42.212077 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.212052    2585 scope.go:117] "RemoveContainer" containerID="00fa7026acfde76514c083f60d8fbb7f44cd720e872c7011c67d86c9e664761c"
Jul 24 22:39:42.214549 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.214529    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"]
Jul 24 22:39:42.217953 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:42.217930    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-brwqp"]
Jul 24 22:39:43.326033 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:43.326003    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" path="/var/lib/kubelet/pods/366c9d07-29fd-4d3d-a457-d6061740fc60/volumes"
Jul 24 22:39:44.203892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:44.203856    2585 generic.go:356] "Generic (PLEG): container finished" podID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerID="f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669" exitCode=0
Jul 24 22:39:44.203892 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:44.203895    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerDied","Data":"f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669"}
Jul 24 22:39:45.209572 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:45.209540    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerStarted","Data":"610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a"}
Jul 24 22:39:45.210074 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:45.209611    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerStarted","Data":"1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31"}
Jul 24 22:39:45.210074 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:45.209840    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:45.230312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:45.230258    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podStartSLOduration=2.623987458 podStartE2EDuration="6.230241796s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.606254338s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:39:45.229778637 +0000 UTC m=+5066.422956574" watchObservedRunningTime="2026-07-24 22:39:45.230241796 +0000 UTC m=+5066.423419732"
Jul 24 22:39:46.213369 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:46.213340    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:46.214468 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:46.214443    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:39:46.790143 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:46.790115    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:39:47.216539 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:47.216502    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:39:52.221857 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:52.221830    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:39:52.222490 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:39:52.222460    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:40:02.223100 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:02.223060    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:40:12.223113 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:12.223068    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:40:22.223002 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:22.222952    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:40:32.223319 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:32.223275    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:40:42.223365 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:42.223282    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:40:49.572376 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:49.572345    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"]
Jul 24 22:40:49.572886 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:49.572615    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" containerID="cri-o://1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31" gracePeriod=30
Jul 24 22:40:49.572886 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:49.572683    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kube-rbac-proxy" containerID="cri-o://610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a" gracePeriod=30
Jul 24 22:40:50.232568 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:50.232530    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:40:50.445185 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:50.445148    2585 generic.go:356] "Generic (PLEG): container finished" podID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerID="610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a" exitCode=2
Jul 24 22:40:50.445373 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:50.445220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerDied","Data":"610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a"}
Jul 24 22:40:52.217097 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:52.217054    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 24 22:40:52.222393 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:52.222365    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 24 22:40:52.910401 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:52.910379    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:40:53.041133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041051    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-rbac-proxy-sar-config\") pod \"a8fec8e9-a1bc-4080-80ae-26d12497faff\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") "
Jul 24 22:40:53.041133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041083    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8fec8e9-a1bc-4080-80ae-26d12497faff-proxy-tls\") pod \"a8fec8e9-a1bc-4080-80ae-26d12497faff\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") "
Jul 24 22:40:53.041367 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041148    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8fec8e9-a1bc-4080-80ae-26d12497faff-kserve-provision-location\") pod \"a8fec8e9-a1bc-4080-80ae-26d12497faff\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") "
Jul 24 22:40:53.041367 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041166    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-cabundle-cert\") pod \"a8fec8e9-a1bc-4080-80ae-26d12497faff\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") "
Jul 24 22:40:53.041367 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041190    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6hl9\" (UniqueName: \"kubernetes.io/projected/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-api-access-m6hl9\") pod \"a8fec8e9-a1bc-4080-80ae-26d12497faff\" (UID: \"a8fec8e9-a1bc-4080-80ae-26d12497faff\") "
Jul 24 22:40:53.041529 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041476    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8fec8e9-a1bc-4080-80ae-26d12497faff" (UID: "a8fec8e9-a1bc-4080-80ae-26d12497faff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:40:53.041529 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041478    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fec8e9-a1bc-4080-80ae-26d12497faff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8fec8e9-a1bc-4080-80ae-26d12497faff" (UID: "a8fec8e9-a1bc-4080-80ae-26d12497faff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:40:53.041600 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.041522    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8fec8e9-a1bc-4080-80ae-26d12497faff" (UID: "a8fec8e9-a1bc-4080-80ae-26d12497faff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:40:53.043416 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.043393    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fec8e9-a1bc-4080-80ae-26d12497faff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8fec8e9-a1bc-4080-80ae-26d12497faff" (UID: "a8fec8e9-a1bc-4080-80ae-26d12497faff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:40:53.043510 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.043457    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-api-access-m6hl9" (OuterVolumeSpecName: "kube-api-access-m6hl9") pod "a8fec8e9-a1bc-4080-80ae-26d12497faff" (UID: "a8fec8e9-a1bc-4080-80ae-26d12497faff"). InnerVolumeSpecName "kube-api-access-m6hl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:40:53.141739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.141711    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8fec8e9-a1bc-4080-80ae-26d12497faff-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:40:53.141739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.141734    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:40:53.141739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.141743    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hl9\" (UniqueName: \"kubernetes.io/projected/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-api-access-m6hl9\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:40:53.141943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.141754    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8fec8e9-a1bc-4080-80ae-26d12497faff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:40:53.141943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.141764    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8fec8e9-a1bc-4080-80ae-26d12497faff-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:40:53.458182 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.458147    2585 generic.go:356] "Generic (PLEG): container finished" podID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerID="1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31" exitCode=0
Jul 24 22:40:53.458535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.458214    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"
Jul 24 22:40:53.458535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.458213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerDied","Data":"1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31"}
Jul 24 22:40:53.458535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.458276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz" event={"ID":"a8fec8e9-a1bc-4080-80ae-26d12497faff","Type":"ContainerDied","Data":"24ae1487190dd29c87014872ef82d8b4badc64ee386050beeecd3a4d0e5f8fb5"}
Jul 24 22:40:53.458535 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.458315    2585 scope.go:117] "RemoveContainer" containerID="610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a"
Jul 24 22:40:53.466253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.466235    2585 scope.go:117] "RemoveContainer" containerID="1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31"
Jul 24 22:40:53.473102 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.473084    2585 scope.go:117] "RemoveContainer" containerID="f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669"
Jul 24 22:40:53.480104 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.480083    2585 scope.go:117] "RemoveContainer" containerID="610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a"
Jul 24 22:40:53.480429 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:40:53.480399    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a\": container with ID starting with 610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a not found: ID does not exist" containerID="610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a"
Jul 24 22:40:53.480533 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.480438    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a"} err="failed to get container status \"610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a\": rpc error: code = NotFound desc = could not find container \"610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a\": container with ID starting with 610ae3bb46c1f6e79f704554b9ad5c2e8e6aff516898a8efed938d4819c7f32a not found: ID does not exist"
Jul 24 22:40:53.480533 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.480461    2585 scope.go:117] "RemoveContainer" containerID="1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31"
Jul 24 22:40:53.480991 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:40:53.480765    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31\": container with ID starting with 1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31 not found: ID does not exist" containerID="1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31"
Jul 24 22:40:53.480991 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.480797    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31"} err="failed to get container status \"1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31\": rpc error: code = NotFound desc = could not find container \"1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31\": container with ID starting with 1a42931aab58e03ff20a9399944ff6759934a6bfbb8d600757f8a8d70a131c31 not found: ID does not exist"
Jul 24 22:40:53.480991 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.480819    2585 scope.go:117] "RemoveContainer" containerID="f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669"
Jul 24 22:40:53.481144 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:40:53.481054    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669\": container with ID starting with f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669 not found: ID does not exist" containerID="f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669"
Jul 24 22:40:53.481144 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.481083    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669"} err="failed to get container status \"f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669\": rpc error: code = NotFound desc = could not find container \"f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669\": container with ID starting with f3bd46715d779665db338459accae48d1694982f59b7e429f3b238d63a48c669 not found: ID does not exist"
Jul 24 22:40:53.482519 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.482499    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"]
Jul 24 22:40:53.487446 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:53.487426    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-m76lz"]
Jul 24 22:40:54.702494 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.702394    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:40:54.731767 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.731735    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"]
Jul 24 22:40:54.731997 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.731985    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kube-rbac-proxy"
Jul 24 22:40:54.732043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.731999    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kube-rbac-proxy"
Jul 24 22:40:54.732043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732010    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kube-rbac-proxy"
Jul 24 22:40:54.732043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732016    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kube-rbac-proxy"
Jul 24 22:40:54.732043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732022    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container"
Jul 24 22:40:54.732043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732027    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container"
Jul 24 22:40:54.732043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732043    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="storage-initializer"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732049    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="storage-initializer"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732057    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kserve-container"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732062    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kserve-container"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732070    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="storage-initializer"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732075    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="storage-initializer"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732112    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kube-rbac-proxy"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732120    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kserve-container"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732125    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="366c9d07-29fd-4d3d-a457-d6061740fc60" containerName="kube-rbac-proxy"
Jul 24 22:40:54.732224 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.732131    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" containerName="kserve-container"
Jul 24 22:40:54.736377 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.736358    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.739284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.739259    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:40:54.739453 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.739361    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 24 22:40:54.739528 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.739494    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 24 22:40:54.740525 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.740503    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:40:54.740525 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.740515    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 24 22:40:54.740739 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.740561    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:40:54.747026 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.747004    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"]
Jul 24 22:40:54.852767 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.852736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.852951 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.852778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.852951 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.852797    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.852951 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.852855    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6xl\" (UniqueName: \"kubernetes.io/projected/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-api-access-sm6xl\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.953378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.953299    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.953378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.953330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.953378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.953351    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6xl\" (UniqueName: \"kubernetes.io/projected/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-api-access-sm6xl\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.953630 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.953407    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.953809 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.953784    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.954074 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.954057    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.955985 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.955963    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.962939 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.962912    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6xl\" (UniqueName: \"kubernetes.io/projected/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-api-access-sm6xl\") pod \"isvc-sklearn-s3-predictor-7d6947b598-66vn6\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:54.994950 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:54.994924    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:40:55.047493 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:55.047466    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:55.173170 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:55.173123    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"]
Jul 24 22:40:55.175378 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:40:55.175344    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768f35f4_bc5a_4240_a3cd_bd2c33f8ec26.slice/crio-a279241aa3898ab729a06ed36b5fa28faea8e9e645fedbdb505d463c137a44c3 WatchSource:0}: Error finding container a279241aa3898ab729a06ed36b5fa28faea8e9e645fedbdb505d463c137a44c3: Status 404 returned error can't find the container with id a279241aa3898ab729a06ed36b5fa28faea8e9e645fedbdb505d463c137a44c3
Jul 24 22:40:55.327552 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:55.327516    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8fec8e9-a1bc-4080-80ae-26d12497faff" path="/var/lib/kubelet/pods/a8fec8e9-a1bc-4080-80ae-26d12497faff/volumes"
Jul 24 22:40:55.468000 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:55.467919    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerStarted","Data":"ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d"}
Jul 24 22:40:55.468000 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:55.467975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerStarted","Data":"a279241aa3898ab729a06ed36b5fa28faea8e9e645fedbdb505d463c137a44c3"}
Jul 24 22:40:56.474018 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:56.473933    2585 generic.go:356] "Generic (PLEG): container finished" podID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerID="ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d" exitCode=0
Jul 24 22:40:56.474390 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:56.474016    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerDied","Data":"ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d"}
Jul 24 22:40:57.480937 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:57.480900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerStarted","Data":"e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632"}
Jul 24 22:40:57.481422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:57.480962    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerStarted","Data":"6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2"}
Jul 24 22:40:57.481422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:57.481033    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:57.504244 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:57.504200    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podStartSLOduration=2.533666024 podStartE2EDuration="3.504187394s" totalImagesPullingTime="0s" totalInitContainerRuntime="970.52137ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:40:57.502924479 +0000 UTC m=+5138.696102464" watchObservedRunningTime="2026-07-24 22:40:57.504187394 +0000 UTC m=+5138.697365328"
Jul 24 22:40:58.485722 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:58.485686    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:40:58.487248 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:58.487220    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:40:59.488765 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:40:59.488721    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:41:04.493931 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:04.493904    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:41:04.494392 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:04.494364    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:41:14.495164 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:14.495116    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:41:24.494907 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:24.494868    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:41:34.494474 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:34.494434    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:41:44.494514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:44.494470    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:41:54.494812 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:41:54.494767    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:42:04.495026 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:04.494982    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:42:08.323686 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:08.323631    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:42:14.864364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:14.864277    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"]
Jul 24 22:42:14.864834 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:14.864560    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" containerID="cri-o://6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2" gracePeriod=30
Jul 24 22:42:14.864834 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:14.864639    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kube-rbac-proxy" containerID="cri-o://e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632" gracePeriod=30
Jul 24 22:42:14.887509 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:14.887450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:42:15.774902 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:15.774864    2585 generic.go:356] "Generic (PLEG): container finished" podID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerID="e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632" exitCode=2
Jul 24 22:42:15.775084 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:15.774910    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerDied","Data":"e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632"}
Jul 24 22:42:16.178108 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:16.178072    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:42:18.323466 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:18.323431    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 24 22:42:19.107392 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.107369    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:42:19.252925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.252840    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kserve-provision-location\") pod \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") "
Jul 24 22:42:19.252925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.252877    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm6xl\" (UniqueName: \"kubernetes.io/projected/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-api-access-sm6xl\") pod \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") "
Jul 24 22:42:19.252925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.252896    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-proxy-tls\") pod \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") "
Jul 24 22:42:19.252925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.252921    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-rbac-proxy-sar-config\") pod \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\" (UID: \"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26\") "
Jul 24 22:42:19.253252 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.253185    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" (UID: "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:42:19.253295 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.253250    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" (UID: "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:42:19.255016 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.254990    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" (UID: "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:42:19.255126 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.255057    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-api-access-sm6xl" (OuterVolumeSpecName: "kube-api-access-sm6xl") pod "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" (UID: "768f35f4-bc5a-4240-a3cd-bd2c33f8ec26"). InnerVolumeSpecName "kube-api-access-sm6xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:42:19.353460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.353436    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:42:19.353460 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.353458    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:42:19.353810 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.353468    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6xl\" (UniqueName: \"kubernetes.io/projected/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-kube-api-access-sm6xl\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:42:19.353810 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.353476    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:42:19.791094 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.791064    2585 generic.go:356] "Generic (PLEG): container finished" podID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerID="6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2" exitCode=0
Jul 24 22:42:19.791253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.791115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerDied","Data":"6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2"}
Jul 24 22:42:19.791253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.791143    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"
Jul 24 22:42:19.791253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.791165    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6" event={"ID":"768f35f4-bc5a-4240-a3cd-bd2c33f8ec26","Type":"ContainerDied","Data":"a279241aa3898ab729a06ed36b5fa28faea8e9e645fedbdb505d463c137a44c3"}
Jul 24 22:42:19.791253 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.791198    2585 scope.go:117] "RemoveContainer" containerID="e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632"
Jul 24 22:42:19.799112 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.799096    2585 scope.go:117] "RemoveContainer" containerID="6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2"
Jul 24 22:42:19.805978 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.805949    2585 scope.go:117] "RemoveContainer" containerID="ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d"
Jul 24 22:42:19.809199 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.809174    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"]
Jul 24 22:42:19.812985 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.812966    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d6947b598-66vn6"]
Jul 24 22:42:19.813418 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.813398    2585 scope.go:117] "RemoveContainer" containerID="e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632"
Jul 24 22:42:19.813707 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:42:19.813678    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632\": container with ID starting with e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632 not found: ID does not exist" containerID="e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632"
Jul 24 22:42:19.813817 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.813706    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632"} err="failed to get container status \"e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632\": rpc error: code = NotFound desc = could not find container \"e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632\": container with ID starting with e0b01a8629423e7a62e6bac2351db4d4901c81596e4900e2f4161da76a3c7632 not found: ID does not exist"
Jul 24 22:42:19.813817 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.813722    2585 scope.go:117] "RemoveContainer" containerID="6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2"
Jul 24 22:42:19.813963 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:42:19.813946    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2\": container with ID starting with 6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2 not found: ID does not exist" containerID="6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2"
Jul 24 22:42:19.814006 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.813967    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2"} err="failed to get container status \"6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2\": rpc error: code = NotFound desc = could not find container \"6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2\": container with ID starting with 6063a4e9325ec2471f4316862756e6cf5443bedc7cba50baf891c8ada2f163b2 not found: ID does not exist"
Jul 24 22:42:19.814006 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.813980    2585 scope.go:117] "RemoveContainer" containerID="ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d"
Jul 24 22:42:19.814202 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:42:19.814188    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d\": container with ID starting with ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d not found: ID does not exist" containerID="ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d"
Jul 24 22:42:19.814252 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:19.814205    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d"} err="failed to get container status \"ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d\": rpc error: code = NotFound desc = could not find container \"ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d\": container with ID starting with ef3fd057e661278ecfa5518a10c894c3857379f16782b750c04e3a823c6f346d not found: ID does not exist"
Jul 24 22:42:21.325970 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:42:21.325939    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" path="/var/lib/kubelet/pods/768f35f4-bc5a-4240-a3cd-bd2c33f8ec26/volumes"
Jul 24 22:43:45.092341 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:45.092304    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:43:46.128734 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.128698    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:43:46.186788 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.186756    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"]
Jul 24 22:43:46.187068 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187045    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container"
Jul 24 22:43:46.187068 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187065    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container"
Jul 24 22:43:46.187068 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187075    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kube-rbac-proxy"
Jul 24 22:43:46.187263 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187081    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kube-rbac-proxy"
Jul 24 22:43:46.187263 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187092    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="storage-initializer"
Jul 24 22:43:46.187263 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187098    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="storage-initializer"
Jul 24 22:43:46.187263 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187166    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kube-rbac-proxy"
Jul 24 22:43:46.187263 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.187178    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="768f35f4-bc5a-4240-a3cd-bd2c33f8ec26" containerName="kserve-container"
Jul 24 22:43:46.190402 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.190384    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.192974 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.192954    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 24 22:43:46.193078 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.192997    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:43:46.193225 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.193211    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 24 22:43:46.193434 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.193413    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:43:46.193552 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.193468    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:43:46.194080 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.194064    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 24 22:43:46.200909 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.200886    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"]
Jul 24 22:43:46.278227 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.278197    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.278357 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.278233    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tff\" (UniqueName: \"kubernetes.io/projected/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-api-access-f8tff\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.278357 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.278273    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.278357 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.278322    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.378964 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.378885    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.378964 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.378936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tff\" (UniqueName: \"kubernetes.io/projected/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-api-access-f8tff\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.379176 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.379016    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.379176 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.379061    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.379176 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:46.379161    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 24 22:43:46.379375 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:46.379237    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls podName:4561a3ef-c9cb-4155-8e56-9da51a8ef5fc nodeName:}" failed. No retries permitted until 2026-07-24 22:43:46.879215902 +0000 UTC m=+5308.072393824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 24 22:43:46.379514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.379494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.379618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.379599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.387816 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.387790    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tff\" (UniqueName: \"kubernetes.io/projected/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-api-access-f8tff\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.560823 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.560792    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:43:46.883215 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.883191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:46.885580 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:46.885559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:47.100737 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:47.100693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:47.233767 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:47.231962    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"]
Jul 24 22:43:48.118651 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:48.118615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" event={"ID":"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc","Type":"ContainerStarted","Data":"fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38"}
Jul 24 22:43:48.118837 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:48.118688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" event={"ID":"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc","Type":"ContainerStarted","Data":"f8ac5f8c610aa2e289144057b76edbacfec6958221fef16f64f591812854e0e2"}
Jul 24 22:43:51.131808 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:51.131777    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/0.log"
Jul 24 22:43:51.132284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:51.131817    2585 generic.go:356] "Generic (PLEG): container finished" podID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerID="fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38" exitCode=1
Jul 24 22:43:51.132284 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:51.131860    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" event={"ID":"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc","Type":"ContainerDied","Data":"fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38"}
Jul 24 22:43:52.137703 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:52.137658    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/0.log"
Jul 24 22:43:52.138050 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:52.137720    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" event={"ID":"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc","Type":"ContainerStarted","Data":"b18667f18a788467b80ef7feb16523266f7e7f12444e085e3f67c2704773484b"}
Jul 24 22:43:56.155545 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.155479    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/1.log"
Jul 24 22:43:56.156026 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.155925    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/0.log"
Jul 24 22:43:56.156026 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.155958    2585 generic.go:356] "Generic (PLEG): container finished" podID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerID="b18667f18a788467b80ef7feb16523266f7e7f12444e085e3f67c2704773484b" exitCode=1
Jul 24 22:43:56.156026 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.155989    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" event={"ID":"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc","Type":"ContainerDied","Data":"b18667f18a788467b80ef7feb16523266f7e7f12444e085e3f67c2704773484b"}
Jul 24 22:43:56.156214 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.156052    2585 scope.go:117] "RemoveContainer" containerID="fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38"
Jul 24 22:43:56.156463 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.156432    2585 scope.go:117] "RemoveContainer" containerID="fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38"
Jul 24 22:43:56.166929 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.166903    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"]
Jul 24 22:43:56.169015 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:56.168970    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_kserve-ci-e2e-test(4561a3ef-c9cb-4155-8e56-9da51a8ef5fc)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc"
Jul 24 22:43:56.169134 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:56.169083    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_kserve-ci-e2e-test_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc_0 in pod sandbox f8ac5f8c610aa2e289144057b76edbacfec6958221fef16f64f591812854e0e2 from index: no such id: 'fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38'" containerID="fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38"
Jul 24 22:43:56.169207 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.169141    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_kserve-ci-e2e-test_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc_0 in pod sandbox f8ac5f8c610aa2e289144057b76edbacfec6958221fef16f64f591812854e0e2 from index: no such id: 'fd2a992931b0f79a41b8229f1d4772f4e193ddb1bc3499827f2ba973597f5f38'"
Jul 24 22:43:56.187839 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:56.187801    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:43:56.252267 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:56.252245    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 24 22:43:56.252395 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:56.252307    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls podName:4561a3ef-c9cb-4155-8e56-9da51a8ef5fc nodeName:}" failed. No retries permitted until 2026-07-24 22:43:56.752288947 +0000 UTC m=+5317.945466864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 24 22:43:56.755714 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:56.755656    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 24 22:43:56.755910 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:56.755754    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls podName:4561a3ef-c9cb-4155-8e56-9da51a8ef5fc nodeName:}" failed. No retries permitted until 2026-07-24 22:43:57.755735516 +0000 UTC m=+5318.948913434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 24 22:43:57.161793 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.161766    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/1.log"
Jul 24 22:43:57.241942 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.241837    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:43:57.274682 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.274642    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"]
Jul 24 22:43:57.279283 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.279261    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.282153 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.282125    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 24 22:43:57.282278 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.282122    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 24 22:43:57.282345 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.282292    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:43:57.293397 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.293372    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"]
Jul 24 22:43:57.314884 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.314863    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/1.log"
Jul 24 22:43:57.317257 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.317240    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:57.358888 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.358860    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.359003 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.358895    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4be4727-4199-4410-9cba-cab88445b9d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.359003 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.358933    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4z8\" (UniqueName: \"kubernetes.io/projected/c4be4727-4199-4410-9cba-cab88445b9d1-kube-api-access-kr4z8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.359095 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.359011    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.359095 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.359084    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460117 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kserve-provision-location\") pod \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") "
Jul 24 22:43:57.460117 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460104    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls\") pod \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") "
Jul 24 22:43:57.460312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460164    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8tff\" (UniqueName: \"kubernetes.io/projected/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-api-access-f8tff\") pod \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") "
Jul 24 22:43:57.460312 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460233    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-rbac-proxy-sar-config\") pod \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\" (UID: \"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc\") "
Jul 24 22:43:57.460427 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4be4727-4199-4410-9cba-cab88445b9d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460427 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460326    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:43:57.460427 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460380    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4z8\" (UniqueName: \"kubernetes.io/projected/c4be4727-4199-4410-9cba-cab88445b9d1-kube-api-access-kr4z8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460427 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460423    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460642 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460470    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460642 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460595    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460642 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:57.460634    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 24 22:43:57.460814 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460645    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:43:57.460814 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460721    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4be4727-4199-4410-9cba-cab88445b9d1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.460814 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.460639    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:43:57.461141 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:43:57.460918    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls podName:c4be4727-4199-4410-9cba-cab88445b9d1 nodeName:}" failed. No retries permitted until 2026-07-24 22:43:57.960893469 +0000 UTC m=+5319.154071386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" (UID: "c4be4727-4199-4410-9cba-cab88445b9d1") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 24 22:43:57.461206 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.461173    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.461324 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.461302    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.462610 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.462580    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:43:57.463024 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.463002    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-api-access-f8tff" (OuterVolumeSpecName: "kube-api-access-f8tff") pod "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" (UID: "4561a3ef-c9cb-4155-8e56-9da51a8ef5fc"). InnerVolumeSpecName "kube-api-access-f8tff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:43:57.469442 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.469418    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4z8\" (UniqueName: \"kubernetes.io/projected/c4be4727-4199-4410-9cba-cab88445b9d1-kube-api-access-kr4z8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.561980 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.561959    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tff\" (UniqueName: \"kubernetes.io/projected/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-api-access-f8tff\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:43:57.561980 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.561980    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:43:57.562105 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.562004    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:43:57.644364 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.644322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:43:57.964371 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.964339    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:57.966914 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:57.966893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:58.166211 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.166184    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t_4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/storage-initializer/1.log"
Jul 24 22:43:58.166585 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.166285    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"
Jul 24 22:43:58.166585 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.166293    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t" event={"ID":"4561a3ef-c9cb-4155-8e56-9da51a8ef5fc","Type":"ContainerDied","Data":"f8ac5f8c610aa2e289144057b76edbacfec6958221fef16f64f591812854e0e2"}
Jul 24 22:43:58.166585 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.166349    2585 scope.go:117] "RemoveContainer" containerID="b18667f18a788467b80ef7feb16523266f7e7f12444e085e3f67c2704773484b"
Jul 24 22:43:58.192164 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.192133    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:43:58.203194 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.203168    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"]
Jul 24 22:43:58.206286 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.206262    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69bdd94894-xbr7t"]
Jul 24 22:43:58.322225 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:58.322192    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"]
Jul 24 22:43:58.325184 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:43:58.325160    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4be4727_4199_4410_9cba_cab88445b9d1.slice/crio-b871651f45b4c2e3d0e20ba5724181c6d27d07e964e1cce763624cd0e51c3d2b WatchSource:0}: Error finding container b871651f45b4c2e3d0e20ba5724181c6d27d07e964e1cce763624cd0e51c3d2b: Status 404 returned error can't find the container with id b871651f45b4c2e3d0e20ba5724181c6d27d07e964e1cce763624cd0e51c3d2b
Jul 24 22:43:59.173259 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:59.173209    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerStarted","Data":"3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324"}
Jul 24 22:43:59.173645 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:59.173315    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerStarted","Data":"b871651f45b4c2e3d0e20ba5724181c6d27d07e964e1cce763624cd0e51c3d2b"}
Jul 24 22:43:59.327320 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:43:59.327277    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" path="/var/lib/kubelet/pods/4561a3ef-c9cb-4155-8e56-9da51a8ef5fc/volumes"
Jul 24 22:44:00.178299 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:00.178265    2585 generic.go:356] "Generic (PLEG): container finished" podID="c4be4727-4199-4410-9cba-cab88445b9d1" containerID="3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324" exitCode=0
Jul 24 22:44:00.178688 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:00.178330    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerDied","Data":"3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324"}
Jul 24 22:44:00.502514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:00.502426    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:44:01.184133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:01.184100    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerStarted","Data":"1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a"}
Jul 24 22:44:01.184523 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:01.184161    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:44:01.184523 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:01.184183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerStarted","Data":"4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4"}
Jul 24 22:44:01.184523 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:01.184228    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:44:01.185728 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:01.185705    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:01.204988 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:01.204941    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podStartSLOduration=3.189240838 podStartE2EDuration="4.204927337s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.015686499s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:44:01.202587716 +0000 UTC m=+5322.395765681" watchObservedRunningTime="2026-07-24 22:44:01.204927337 +0000 UTC m=+5322.398105337"
Jul 24 22:44:02.188945 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:02.188904    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:07.193471 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:07.193441    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:44:07.194098 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:07.194065    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:17.194154 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:17.194114    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:27.194683 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:27.194629    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:37.194362 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:37.194325    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:47.194612 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:47.194572    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:44:57.194221 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:44:57.194179    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:45:07.194025 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:07.193987    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:45:11.327975 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:11.327783    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:45:17.326335 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:17.326262    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"]
Jul 24 22:45:17.326826 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:17.326485    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" containerID="cri-o://4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4" gracePeriod=30
Jul 24 22:45:17.326826 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:17.326513    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kube-rbac-proxy" containerID="cri-o://1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a" gracePeriod=30
Jul 24 22:45:17.349339 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:17.349312    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:45:17.462826 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:17.462797    2585 generic.go:356] "Generic (PLEG): container finished" podID="c4be4727-4199-4410-9cba-cab88445b9d1" containerID="1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a" exitCode=2
Jul 24 22:45:17.462925 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:17.462846    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerDied","Data":"1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a"}
Jul 24 22:45:18.338491 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:18.338455    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:45:18.893986 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:18.893950    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:45:21.322587 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.322546    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 24 22:45:21.671722 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.671699    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:45:21.763878 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.763848    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-cabundle-cert\") pod \"c4be4727-4199-4410-9cba-cab88445b9d1\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") "
Jul 24 22:45:21.764055 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.763897    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4be4727-4199-4410-9cba-cab88445b9d1-kserve-provision-location\") pod \"c4be4727-4199-4410-9cba-cab88445b9d1\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") "
Jul 24 22:45:21.764055 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.763927    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-kube-rbac-proxy-sar-config\") pod \"c4be4727-4199-4410-9cba-cab88445b9d1\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") "
Jul 24 22:45:21.764055 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.764011    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr4z8\" (UniqueName: \"kubernetes.io/projected/c4be4727-4199-4410-9cba-cab88445b9d1-kube-api-access-kr4z8\") pod \"c4be4727-4199-4410-9cba-cab88445b9d1\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") "
Jul 24 22:45:21.764055 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.764043    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls\") pod \"c4be4727-4199-4410-9cba-cab88445b9d1\" (UID: \"c4be4727-4199-4410-9cba-cab88445b9d1\") "
Jul 24 22:45:21.764297 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.764272    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4be4727-4199-4410-9cba-cab88445b9d1" (UID: "c4be4727-4199-4410-9cba-cab88445b9d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:45:21.764348 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.764271    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4be4727-4199-4410-9cba-cab88445b9d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4be4727-4199-4410-9cba-cab88445b9d1" (UID: "c4be4727-4199-4410-9cba-cab88445b9d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:45:21.764389 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.764368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4be4727-4199-4410-9cba-cab88445b9d1" (UID: "c4be4727-4199-4410-9cba-cab88445b9d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:45:21.766392 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.766368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4be4727-4199-4410-9cba-cab88445b9d1-kube-api-access-kr4z8" (OuterVolumeSpecName: "kube-api-access-kr4z8") pod "c4be4727-4199-4410-9cba-cab88445b9d1" (UID: "c4be4727-4199-4410-9cba-cab88445b9d1"). InnerVolumeSpecName "kube-api-access-kr4z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:45:21.766499 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.766439    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4be4727-4199-4410-9cba-cab88445b9d1" (UID: "c4be4727-4199-4410-9cba-cab88445b9d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:45:21.865473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.865416    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4z8\" (UniqueName: \"kubernetes.io/projected/c4be4727-4199-4410-9cba-cab88445b9d1-kube-api-access-kr4z8\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:45:21.865473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.865439    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4be4727-4199-4410-9cba-cab88445b9d1-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:45:21.865473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.865448    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:45:21.865473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.865456    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4be4727-4199-4410-9cba-cab88445b9d1-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:45:21.865473 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:21.865464    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4be4727-4199-4410-9cba-cab88445b9d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:45:22.482280 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.482240    2585 generic.go:356] "Generic (PLEG): container finished" podID="c4be4727-4199-4410-9cba-cab88445b9d1" containerID="4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4" exitCode=0
Jul 24 22:45:22.482736 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.482296    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerDied","Data":"4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4"}
Jul 24 22:45:22.482736 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.482307    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"
Jul 24 22:45:22.482736 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.482359    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8" event={"ID":"c4be4727-4199-4410-9cba-cab88445b9d1","Type":"ContainerDied","Data":"b871651f45b4c2e3d0e20ba5724181c6d27d07e964e1cce763624cd0e51c3d2b"}
Jul 24 22:45:22.482736 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.482392    2585 scope.go:117] "RemoveContainer" containerID="1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a"
Jul 24 22:45:22.493347 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.492269    2585 scope.go:117] "RemoveContainer" containerID="4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4"
Jul 24 22:45:22.500868 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.500837    2585 scope.go:117] "RemoveContainer" containerID="3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324"
Jul 24 22:45:22.507971 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.507952    2585 scope.go:117] "RemoveContainer" containerID="1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a"
Jul 24 22:45:22.508242 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:22.508220    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a\": container with ID starting with 1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a not found: ID does not exist" containerID="1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a"
Jul 24 22:45:22.508327 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.508250    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a"} err="failed to get container status \"1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a\": rpc error: code = NotFound desc = could not find container \"1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a\": container with ID starting with 1e986cfd83d5e7dde7ac1ee5d95f92deeadabdbcca2c737316112c8458bca65a not found: ID does not exist"
Jul 24 22:45:22.508327 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.508272    2585 scope.go:117] "RemoveContainer" containerID="4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4"
Jul 24 22:45:22.508537 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:22.508517    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4\": container with ID starting with 4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4 not found: ID does not exist" containerID="4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4"
Jul 24 22:45:22.508628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.508544    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4"} err="failed to get container status \"4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4\": rpc error: code = NotFound desc = could not find container \"4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4\": container with ID starting with 4f234748009a1f9cfb5f05c251455d4ff7c69053e0d7a4bef6221d972327f0f4 not found: ID does not exist"
Jul 24 22:45:22.508628 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.508576    2585 scope.go:117] "RemoveContainer" containerID="3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324"
Jul 24 22:45:22.508868 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.508845    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"]
Jul 24 22:45:22.508868 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:22.508854    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324\": container with ID starting with 3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324 not found: ID does not exist" containerID="3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324"
Jul 24 22:45:22.509014 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.508880    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324"} err="failed to get container status \"3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324\": rpc error: code = NotFound desc = could not find container \"3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324\": container with ID starting with 3847eb97106490db6c944f5e27dd51798133de892ef652c8a5750a422fd4a324 not found: ID does not exist"
Jul 24 22:45:22.515514 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:22.515493    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-746f6cb4f8-j9jc8"]
Jul 24 22:45:23.326283 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:23.326251    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" path="/var/lib/kubelet/pods/c4be4727-4199-4410-9cba-cab88445b9d1/volumes"
Jul 24 22:45:28.385430 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:28.385400    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:45:29.447599 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.447561    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:45:29.465144 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465114    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"]
Jul 24 22:45:29.465368 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465355    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerName="storage-initializer"
Jul 24 22:45:29.465411 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465369    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerName="storage-initializer"
Jul 24 22:45:29.465411 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465380    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="storage-initializer"
Jul 24 22:45:29.465411 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465386    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="storage-initializer"
Jul 24 22:45:29.465411 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465394    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerName="storage-initializer"
Jul 24 22:45:29.465411 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465400    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerName="storage-initializer"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465414    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465420    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465427    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kube-rbac-proxy"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465432    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kube-rbac-proxy"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465471    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kserve-container"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465478    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerName="storage-initializer"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465483    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4be4727-4199-4410-9cba-cab88445b9d1" containerName="kube-rbac-proxy"
Jul 24 22:45:29.465618 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.465489    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4561a3ef-c9cb-4155-8e56-9da51a8ef5fc" containerName="storage-initializer"
Jul 24 22:45:29.470179 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.470163    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.472517 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.472493    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 24 22:45:29.472744 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.472728    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 24 22:45:29.472816 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.472755    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 22:45:29.472816 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.472785    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 24 22:45:29.473820 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.473803    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 22:45:29.473924 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.473803    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 22:45:29.474200 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.474180    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lswxz\""
Jul 24 22:45:29.478613 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.478591    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"]
Jul 24 22:45:29.516979 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.516948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6938eeb3-0217-4b71-b7d9-b8b161429913-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.517124 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.517010    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.517124 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.517050    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4vb\" (UniqueName: \"kubernetes.io/projected/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-api-access-zn4vb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.517220 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.517124    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.517220 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.517155    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.618294 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.618256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4vb\" (UniqueName: \"kubernetes.io/projected/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-api-access-zn4vb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.618438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.618308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.618438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.618336    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.618438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.618371    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6938eeb3-0217-4b71-b7d9-b8b161429913-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.618438 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.618419    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.618703 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:29.618434    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 24 22:45:29.618703 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:29.618504    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls podName:6938eeb3-0217-4b71-b7d9-b8b161429913 nodeName:}" failed. No retries permitted until 2026-07-24 22:45:30.118482131 +0000 UTC m=+5411.311660050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 24 22:45:29.618917 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.618887    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6938eeb3-0217-4b71-b7d9-b8b161429913-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.619181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.619163    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.619250 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.619200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:29.626945 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:29.626921    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4vb\" (UniqueName: \"kubernetes.io/projected/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-api-access-zn4vb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:30.122189 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:30.122153    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:30.122369 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:30.122300    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 24 22:45:30.122419 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:45:30.122374    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls podName:6938eeb3-0217-4b71-b7d9-b8b161429913 nodeName:}" failed. No retries permitted until 2026-07-24 22:45:31.12235876 +0000 UTC m=+5412.315536675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 24 22:45:30.180150 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:30.180098    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:45:31.129519 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:31.129485    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:31.132043 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:31.132016    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:31.281158 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:31.281111    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:31.413609 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:31.413583    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"]
Jul 24 22:45:31.415415 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:45:31.415390    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6938eeb3_0217_4b71_b7d9_b8b161429913.slice/crio-dffcc5b2ceb2adb416f466f3c959a147c32388b90d75142ebf350ced334f551b WatchSource:0}: Error finding container dffcc5b2ceb2adb416f466f3c959a147c32388b90d75142ebf350ced334f551b: Status 404 returned error can't find the container with id dffcc5b2ceb2adb416f466f3c959a147c32388b90d75142ebf350ced334f551b
Jul 24 22:45:31.517865 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:31.517834    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerStarted","Data":"a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511"}
Jul 24 22:45:31.517972 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:31.517904    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerStarted","Data":"dffcc5b2ceb2adb416f466f3c959a147c32388b90d75142ebf350ced334f551b"}
Jul 24 22:45:32.523305 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:32.523277    2585 generic.go:356] "Generic (PLEG): container finished" podID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerID="a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511" exitCode=0
Jul 24 22:45:32.523699 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:32.523365    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerDied","Data":"a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511"}
Jul 24 22:45:33.529244 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:33.529200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerStarted","Data":"b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b"}
Jul 24 22:45:33.529624 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:33.529275    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerStarted","Data":"61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304"}
Jul 24 22:45:33.529624 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:33.529411    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:34.532957 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:34.532927    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:34.534204 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:34.534179    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:45:35.536590 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:35.536536    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:45:40.541479 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:40.541449    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:45:40.541943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:40.541917    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:45:40.562455 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:40.562402    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podStartSLOduration=10.56790144 podStartE2EDuration="11.562389491s" totalImagesPullingTime="0s" totalInitContainerRuntime="994.488051ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:45:33.54873641 +0000 UTC m=+5414.741914372" watchObservedRunningTime="2026-07-24 22:45:40.562389491 +0000 UTC m=+5421.755567427"
Jul 24 22:45:50.542719 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:45:50.542682    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:46:00.542864 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:00.542824    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:46:10.542132 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:10.542092    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:46:20.542232 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:20.542191    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:46:30.542780 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:30.542743    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:46:40.542278 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:40.542185    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:46:50.542378 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:50.542340    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:46:59.550786 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:59.550752    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"]
Jul 24 22:46:59.551232 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:59.550979    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" containerID="cri-o://61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304" gracePeriod=30
Jul 24 22:46:59.551232 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:59.551026    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kube-rbac-proxy" containerID="cri-o://b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b" gracePeriod=30
Jul 24 22:46:59.563853 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:59.563832    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:59.841373 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:59.841289    2585 generic.go:356] "Generic (PLEG): container finished" podID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerID="b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b" exitCode=2
Jul 24 22:46:59.841373 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:46:59.841340    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerDied","Data":"b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b"}
Jul 24 22:47:00.536899 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:00.536856    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 24 22:47:00.541942 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:00.541905    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 24 22:47:00.578449 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:00.578417    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:01.060936 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:01.060888    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:03.801413 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.801387    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:47:03.863100 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.863008    2585 generic.go:356] "Generic (PLEG): container finished" podID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerID="61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304" exitCode=0
Jul 24 22:47:03.863251 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.863120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerDied","Data":"61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304"}
Jul 24 22:47:03.863251 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.863225    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv" event={"ID":"6938eeb3-0217-4b71-b7d9-b8b161429913","Type":"ContainerDied","Data":"dffcc5b2ceb2adb416f466f3c959a147c32388b90d75142ebf350ced334f551b"}
Jul 24 22:47:03.863361 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.863274    2585 scope.go:117] "RemoveContainer" containerID="b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b"
Jul 24 22:47:03.863501 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.863483    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"
Jul 24 22:47:03.872958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.872936    2585 scope.go:117] "RemoveContainer" containerID="61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304"
Jul 24 22:47:03.881196 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.881179    2585 scope.go:117] "RemoveContainer" containerID="a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511"
Jul 24 22:47:03.888295 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.888279    2585 scope.go:117] "RemoveContainer" containerID="b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b"
Jul 24 22:47:03.888568 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:47:03.888549    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b\": container with ID starting with b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b not found: ID does not exist" containerID="b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b"
Jul 24 22:47:03.888651 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.888588    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b"} err="failed to get container status \"b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b\": rpc error: code = NotFound desc = could not find container \"b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b\": container with ID starting with b6e1f11e2b002f0d2c1a411c828273bbce19d36ff493e8e0b76807ee8b37671b not found: ID does not exist"
Jul 24 22:47:03.888651 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.888613    2585 scope.go:117] "RemoveContainer" containerID="61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304"
Jul 24 22:47:03.888929 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:47:03.888904    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304\": container with ID starting with 61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304 not found: ID does not exist" containerID="61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304"
Jul 24 22:47:03.888971 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.888938    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304"} err="failed to get container status \"61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304\": rpc error: code = NotFound desc = could not find container \"61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304\": container with ID starting with 61c9b0c5e5a54d42114a85e67b92844f404f3e093e3e514ff1105d8451118304 not found: ID does not exist"
Jul 24 22:47:03.888971 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.888960    2585 scope.go:117] "RemoveContainer" containerID="a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511"
Jul 24 22:47:03.889204 ip-10-0-143-243 kubenswrapper[2585]: E0724 22:47:03.889183    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511\": container with ID starting with a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511 not found: ID does not exist" containerID="a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511"
Jul 24 22:47:03.889246 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.889210    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511"} err="failed to get container status \"a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511\": rpc error: code = NotFound desc = could not find container \"a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511\": container with ID starting with a533381bc4c8e81ea3143cab73d4992d25b97b7d0aa98fc6b84116cfbe142511 not found: ID does not exist"
Jul 24 22:47:03.900113 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900097    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-cabundle-cert\") pod \"6938eeb3-0217-4b71-b7d9-b8b161429913\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") "
Jul 24 22:47:03.900181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900157    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn4vb\" (UniqueName: \"kubernetes.io/projected/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-api-access-zn4vb\") pod \"6938eeb3-0217-4b71-b7d9-b8b161429913\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") "
Jul 24 22:47:03.900181 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900177    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-rbac-proxy-sar-config\") pod \"6938eeb3-0217-4b71-b7d9-b8b161429913\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") "
Jul 24 22:47:03.900287 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900199    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") pod \"6938eeb3-0217-4b71-b7d9-b8b161429913\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") "
Jul 24 22:47:03.900287 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900238    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6938eeb3-0217-4b71-b7d9-b8b161429913-kserve-provision-location\") pod \"6938eeb3-0217-4b71-b7d9-b8b161429913\" (UID: \"6938eeb3-0217-4b71-b7d9-b8b161429913\") "
Jul 24 22:47:03.900552 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900528    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6938eeb3-0217-4b71-b7d9-b8b161429913" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:47:03.900649 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900562    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6938eeb3-0217-4b71-b7d9-b8b161429913" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:47:03.900649 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.900568    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6938eeb3-0217-4b71-b7d9-b8b161429913-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6938eeb3-0217-4b71-b7d9-b8b161429913" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:47:03.902194 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.902174    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-api-access-zn4vb" (OuterVolumeSpecName: "kube-api-access-zn4vb") pod "6938eeb3-0217-4b71-b7d9-b8b161429913" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913"). InnerVolumeSpecName "kube-api-access-zn4vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:47:03.902320 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:03.902299    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6938eeb3-0217-4b71-b7d9-b8b161429913" (UID: "6938eeb3-0217-4b71-b7d9-b8b161429913"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:47:04.001095 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.001073    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6938eeb3-0217-4b71-b7d9-b8b161429913-kserve-provision-location\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:47:04.001095 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.001094    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-cabundle-cert\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:47:04.001223 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.001104    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4vb\" (UniqueName: \"kubernetes.io/projected/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-api-access-zn4vb\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:47:04.001223 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.001113    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6938eeb3-0217-4b71-b7d9-b8b161429913-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:47:04.001223 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.001121    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6938eeb3-0217-4b71-b7d9-b8b161429913-proxy-tls\") on node \"ip-10-0-143-243.ec2.internal\" DevicePath \"\""
Jul 24 22:47:04.185821 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.185795    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"]
Jul 24 22:47:04.189314 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:04.189293    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-5c46f6b55c-2jxlv"]
Jul 24 22:47:05.326994 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:05.326964    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" path="/var/lib/kubelet/pods/6938eeb3-0217-4b71-b7d9-b8b161429913/volumes"
Jul 24 22:47:10.626646 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:10.626611    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:12.354347 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:12.354311    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:12.456084 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:12.456053    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:46.898005 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:46.897968    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:47.984684 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:47.984638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:49.871568 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:49.871537    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7kk4m_67cabf5a-03ac-492a-9bc0-79d3c1d8dcdf/global-pull-secret-syncer/0.log"
Jul 24 22:47:49.998349 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:49.998318    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log"
Jul 24 22:47:50.211306 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:50.211226    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-243.ec2.internal_c1bea1286bfe6de533a5546f72ba9243/haproxy/0.log"
Jul 24 22:47:52.573884 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:52.573851    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:52.625875 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:52.625838    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2hw84_5452a9bc-1899-4ec2-ae0b-78039a6722e8/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:54.881650 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:54.881622    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-26llm_227c9960-a563-481c-999d-b8eeb54abe48/node-exporter/0.log"
Jul 24 22:47:54.907580 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:54.907554    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-26llm_227c9960-a563-481c-999d-b8eeb54abe48/kube-rbac-proxy/0.log"
Jul 24 22:47:54.931457 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:54.931435    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-26llm_227c9960-a563-481c-999d-b8eeb54abe48/init-textfile/0.log"
Jul 24 22:47:56.995587 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:56.995560    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-qpt4v_cbefa37b-bdef-4291-8ca3-2271f8f5b02d/networking-console-plugin/0.log"
Jul 24 22:47:58.365451 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365413    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"]
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365716    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365729    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365743    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kube-rbac-proxy"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365749    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kube-rbac-proxy"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365761    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="storage-initializer"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365767    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="storage-initializer"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365816    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kserve-container"
Jul 24 22:47:58.365860 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.365824    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6938eeb3-0217-4b71-b7d9-b8b161429913" containerName="kube-rbac-proxy"
Jul 24 22:47:58.368622 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.368601    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.371157 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.371136    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcx84\"/\"kube-root-ca.crt\""
Jul 24 22:47:58.371277 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.371174    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcx84\"/\"openshift-service-ca.crt\""
Jul 24 22:47:58.372349 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.372323    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mcx84\"/\"default-dockercfg-zm8wh\""
Jul 24 22:47:58.376943 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.376919    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"]
Jul 24 22:47:58.490422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.490386    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-proc\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.490422 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.490424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-lib-modules\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.490654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.490445    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-podres\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.490654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.490495    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-sys\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.490654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.490557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmqm\" (UniqueName: \"kubernetes.io/projected/8c9346f1-af08-4e23-90e0-b0a18dec26e6-kube-api-access-skmqm\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.591801 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.591764    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmqm\" (UniqueName: \"kubernetes.io/projected/8c9346f1-af08-4e23-90e0-b0a18dec26e6-kube-api-access-skmqm\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.591960 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.591814    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-proc\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.591960 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.591889    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-proc\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.591960 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.591928    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-lib-modules\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.592070 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.591965    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-podres\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.592070 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.591990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-sys\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.592133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.592079    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-podres\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.592133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.592087    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-lib-modules\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.592133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.592080    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c9346f1-af08-4e23-90e0-b0a18dec26e6-sys\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.599756 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.599736    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmqm\" (UniqueName: \"kubernetes.io/projected/8c9346f1-af08-4e23-90e0-b0a18dec26e6-kube-api-access-skmqm\") pod \"perf-node-gather-daemonset-g6v5p\" (UID: \"8c9346f1-af08-4e23-90e0-b0a18dec26e6\") " pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.678992 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.678958    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:58.804871 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:58.804840    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"]
Jul 24 22:47:58.808000 ip-10-0-143-243 kubenswrapper[2585]: W0724 22:47:58.807974    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c9346f1_af08_4e23_90e0_b0a18dec26e6.slice/crio-842157f261a952dc52f3fd8035ced55061714460ec977ca7b637100af9f46d03 WatchSource:0}: Error finding container 842157f261a952dc52f3fd8035ced55061714460ec977ca7b637100af9f46d03: Status 404 returned error can't find the container with id 842157f261a952dc52f3fd8035ced55061714460ec977ca7b637100af9f46d03
Jul 24 22:47:59.062800 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.062713    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p" event={"ID":"8c9346f1-af08-4e23-90e0-b0a18dec26e6","Type":"ContainerStarted","Data":"eba72cb4e5544f5ffc8c1ccefa791f7761ae1b446946330688a04ebea5ddd705"}
Jul 24 22:47:59.062800 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.062766    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p" event={"ID":"8c9346f1-af08-4e23-90e0-b0a18dec26e6","Type":"ContainerStarted","Data":"842157f261a952dc52f3fd8035ced55061714460ec977ca7b637100af9f46d03"}
Jul 24 22:47:59.063010 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.062856    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:47:59.078496 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.078451    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p" podStartSLOduration=1.0784382240000001 podStartE2EDuration="1.078438224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:59.078317781 +0000 UTC m=+5560.271495743" watchObservedRunningTime="2026-07-24 22:47:59.078438224 +0000 UTC m=+5560.271616161"
Jul 24 22:47:59.467485 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.467455    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vkrgq_37d7e99c-5d52-4c53-a7a2-e46f3910261a/dns/0.log"
Jul 24 22:47:59.488911 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.488885    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vkrgq_37d7e99c-5d52-4c53-a7a2-e46f3910261a/kube-rbac-proxy/0.log"
Jul 24 22:47:59.535133 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:47:59.535105    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5bml_392c898d-7923-488d-b0c1-1da116ad9ec0/dns-node-resolver/0.log"
Jul 24 22:48:00.044214 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:00.044185    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66568f7d4b-gvv94_5306f000-7fcd-4bf9-8e40-60f294049351/registry/0.log"
Jul 24 22:48:00.148484 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:00.148449    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tq265_df23e468-1389-4fbc-9e50-06d5d97d3e1c/node-ca/0.log"
Jul 24 22:48:01.278891 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:01.278859    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fczl6_551c542a-9f19-47f8-b41e-2dd2f21f674d/serve-healthcheck-canary/0.log"
Jul 24 22:48:01.816399 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:01.816368    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xc5zd_70b90561-8565-4a7f-a290-73b7dd0a1f60/kube-rbac-proxy/0.log"
Jul 24 22:48:01.839044 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:01.839013    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xc5zd_70b90561-8565-4a7f-a290-73b7dd0a1f60/exporter/0.log"
Jul 24 22:48:01.861160 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:01.861132    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xc5zd_70b90561-8565-4a7f-a290-73b7dd0a1f60/extractor/0.log"
Jul 24 22:48:04.139268 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:04.139235    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-664687df58-pr42s_bddba108-8a4b-4745-94cd-7808b5aa308b/manager/0.log"
Jul 24 22:48:04.489461 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:04.489379    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-76kg8_aebee1e6-2c9b-4cb6-9a69-ad6d209f5153/manager/0.log"
Jul 24 22:48:05.075917 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:05.075884    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mcx84/perf-node-gather-daemonset-g6v5p"
Jul 24 22:48:08.933217 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:08.933191    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-lqpct_be4d127f-5b1f-412b-8ffe-dc2b2e940908/migrator/0.log"
Jul 24 22:48:08.956442 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:08.956423    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-lqpct_be4d127f-5b1f-412b-8ffe-dc2b2e940908/graceful-termination/0.log"
Jul 24 22:48:10.215092 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.215060    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/kube-multus-additional-cni-plugins/0.log"
Jul 24 22:48:10.235527 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.235505    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/egress-router-binary-copy/0.log"
Jul 24 22:48:10.255179 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.255157    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/cni-plugins/0.log"
Jul 24 22:48:10.275099 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.275082    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/bond-cni-plugin/0.log"
Jul 24 22:48:10.295479 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.295431    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/routeoverride-cni/0.log"
Jul 24 22:48:10.318789 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.318769    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/whereabouts-cni-bincopy/0.log"
Jul 24 22:48:10.338958 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.338941    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8plsn_4163186b-2c9e-42d1-ba75-0f9d45be9ff8/whereabouts-cni/0.log"
Jul 24 22:48:10.674235 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.674210    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gxvfx_5c551ea5-6857-4c62-ab6c-dd0d9182a9dc/kube-multus/0.log"
Jul 24 22:48:10.760579 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.760548    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5sl4p_694177d1-b8fa-43ec-9ad0-dbd7aa96111f/network-metrics-daemon/0.log"
Jul 24 22:48:10.779491 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:10.779429    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5sl4p_694177d1-b8fa-43ec-9ad0-dbd7aa96111f/kube-rbac-proxy/0.log"
Jul 24 22:48:11.593643 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.593617    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/ovn-controller/0.log"
Jul 24 22:48:11.610912 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.610870    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/ovn-acl-logging/0.log"
Jul 24 22:48:11.632226 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.632203    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/kube-rbac-proxy-node/0.log"
Jul 24 22:48:11.651189 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.651165    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/kube-rbac-proxy-ovn-metrics/0.log"
Jul 24 22:48:11.670360 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.670336    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/northd/0.log"
Jul 24 22:48:11.690002 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.689971    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/nbdb/0.log"
Jul 24 22:48:11.709998 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.709976    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/sbdb/0.log"
Jul 24 22:48:11.758749 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:11.758728    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6nm8n_980a987a-0aa5-4f5f-8c93-ab877c736e8e/ovnkube-controller/0.log"
Jul 24 22:48:13.190654 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:13.190624    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-n6hjf_2dae7df5-d719-4b5b-83c5-be9c6af6b24c/network-check-target-container/0.log"
Jul 24 22:48:14.123227 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:14.123196    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-v7hcf_c187ee75-0aff-41ff-be95-a05e6ca2cff3/iptables-alerter/0.log"
Jul 24 22:48:14.766419 ip-10-0-143-243 kubenswrapper[2585]: I0724 22:48:14.766393    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-l7w94_260c4dbc-58c6-4a0c-984d-9b8bce3f558a/tuned/0.log"